Starting in 1.1.0beta8 logstash begins to intermittently hang during grok parsing holding a thread at 100% of a CPU core. After several minutes or so the thread will die off before hanging again. During the thread hang nothing is processed.
Beta7 does not appear to have this behavior. Over 2 million lines processed without a halt.
Attached is a ps aux -L and a jstack of the process. It's hanging on LWP 17000; 4268 in hex under jstack. Also attached is my collector.conf and patterns file. It is the cfserver or cfusion pattern causing this, though I haven't been able to identify what specific line(s) are causing the hang. If you need more info or some sample log lines please contact me on IRC: colon_D #logstash.
Thanks to Greg for point out this issue to me on IRC yesterday!
I started seeing lockups when I installed 1.1.0 on our production log server. As Greg described, it runs for a short period of time and locks up. Web interface does not respond and new log entries do not get processed. I'd restart logstash and it would start working again to just lock up again a short time later.
I can confirm that going back to beta7 worked fine. I went from only being able to process a few hundred to a thousand entries on 1.1.0 to processing 2494122 overnight (after filtering a lot).
Dup of (which has an example config + input that reproduces this bug)
WIll continue this thread on
History of that is now fixed
Some grok patterns have been fixed, and there's now a filter thread watchdog. If this breaks again in 1.1.1, please file a new ticket w/your grok config and the last few log lines logstash emits before crashing.