Slow startup time with lots of grok patterns

Description

I use a large list of grok filters to parse request logs and attempt to tag each of the methods to a particular resource.

I have a grok pattern file with 79 patterns and the same number of individual grok filters which match each of those patterns and add tags to the message to mark it and prevent it being matched again.

While starting up the logstash agent with the '-v' flag I've noticed that the following output happens for every grok filter instance:

This causes the startup time of my agent to be upwards of three minutes.

Reglardless of my crazy use of the grok filter (which I'm more than happy to be corrected on), is it possible to cache each of the grok patterns during startup rather than load each and every file multiple times?

Activity

Show:
Former user
November 25, 2013, 7:18 PM

Do you need more information about this issue?

Justin Lambert
January 27, 2014, 12:45 PM

I was just noticing this with my config as well - 10 minute + startup time and I suspect a large amount of it is my grok patterns. I see the same thing, loading patters every time it is used in the config. LS 1.3.3

Phil Helmer
February 7, 2014, 1:23 AM

Using 1.3.3 and config of 9 grok filters takes close to 40 seconds before LS starts processing events. This is not using any custom patterns. Also, the command line is the default agent, not altering any of the default java memory settings. > 2GB free memory on the machine before starting LS.

Former user
November 24, 2014, 5:20 PM

I can confirm this issue has gone away after upgrading to 1.4.2.

Assignee

Logstash Developers

Reporter

Former user

Labels

Affects versions

Configure