Grok parsing causes thread hang at 100%CPU

Description

Starting in 1.1.0beta8 logstash begins to intermittently hang during grok parsing holding a thread at 100% of a CPU core. After several minutes or so the thread will die off before hanging again. During the thread hang nothing is processed.

Beta7 does not appear to have this behavior. Over 2 million lines processed without a halt.

Attached is a ps aux -L and a jstack of the process. It's hanging on LWP 17000; 4268 in hex under jstack. Also attached is my collector.conf and patterns file. It is the cfserver or cfusion pattern causing this, though I haven't been able to identify what specific line(s) are causing the hang. If you need more info or some sample log lines please contact me on IRC: colon_D #logstash.

Activity

Show:
Andy Botting
February 17, 2012, 8:35 AM

Thanks to Greg for point out this issue to me on IRC yesterday!

I started seeing lockups when I installed 1.1.0 on our production log server. As Greg described, it runs for a short period of time and locks up. Web interface does not respond and new log entries do not get processed. I'd restart logstash and it would start working again to just lock up again a short time later.

I can confirm that going back to beta7 worked fine. I went from only being able to process a few hundred to a thousand entries on 1.1.0 to processing 2494122 overnight (after filtering a lot).

Jordan Sissel
March 24, 2012, 3:11 AM

Dup of (which has an example config + input that reproduces this bug)

Jordan Sissel
March 24, 2012, 3:11 AM

WIll continue this thread on

Philippe Weber
June 22, 2012, 2:46 PM

CanBeClosed
History of that is now fixed

PeteF
June 22, 2012, 3:33 PM

Some grok patterns have been fixed, and there's now a filter thread watchdog. If this breaks again in 1.1.1, please file a new ticket w/your grok config and the last few log lines logstash emits before crashing.

Assignee

Logstash Developers

Reporter

Greg Dallavalle

Labels

Fix versions

Affects versions

Configure