I use a large list of grok filters to parse request logs and attempt to tag each of the methods to a particular resource.
I have a grok pattern file with 79 patterns and the same number of individual grok filters which match each of those patterns and add tags to the message to mark it and prevent it being matched again.
While starting up the logstash agent with the '-v' flag I've noticed that the following output happens for every grok filter instance:
This causes the startup time of my agent to be upwards of three minutes.
Reglardless of my crazy use of the grok filter (which I'm more than happy to be corrected on), is it possible to cache each of the grok patterns during startup rather than load each and every file multiple times?
Do you need more information about this issue?
I was just noticing this with my config as well - 10 minute + startup time and I suspect a large amount of it is my grok patterns. I see the same thing, loading patters every time it is used in the config. LS 1.3.3
Using 1.3.3 and config of 9 grok filters takes close to 40 seconds before LS starts processing events. This is not using any custom patterns. Also, the command line is the default agent, not altering any of the default java memory settings. > 2GB free memory on the machine before starting LS.
I can confirm this issue has gone away after upgrading to 1.4.2.