Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Logstash thread watchdog timeout crash

Description

Logstash crashes with my nginx grok filter every few minutes or hours. Before my grok filter was simpler and did not contain more than two patterns...

I have attached the crash log and my nginx pattern.

Logstash receives a few million messages per day in our setup.

Attachments

2
  • 05 Apr 2013, 07:51 AM
  • 05 Apr 2013, 07:51 AM

Gliffy Diagrams

Activity

Show:

Aaron Sterr July 4, 2013 at 11:43 PM
Edited

This happened to me as well when I added a fourth pattern to one of my grok filters.

Here is the grok filter:

grok { type => "lhdr" tags => [ 'lhdr', 'lhdrlog' ] pattern => '%{NUMBER} \[%{DATA:thread}\] %{WORD:priority} %{DATA:class} %{GREEDYDATA:message}' pattern => '\[%{TIMESTAMP_ISO8601:timestamp}\]\[%{WORD:priority}\]\[%{DATA:thread}\]\[%{DATA:class}\] %{GREEDYDATA:message}' pattern => '\[(?<timestamp>%{SYSLOGTIMESTAMP} %{DATA})\] \[%{DATA:thread}\] %{GREEDYDATA:message}' pattern => '%{DATA:priority} (?<timestamp>%{DATA}+) \[%{DATA:thread}\] %{DATA:class} %{GREEDYDATA:message}' }

Removing the fourth pattern fixes the problem.

For reference, I am currently indexing ~20 million messages per day. It has been holding up so far.

Actually – I found another work around. I changed the pattern to this and now I am working:

%{DATA:priority} (?<timestamp>.+) \[%{DATA:thread}\] %{DATA:class} %{GREEDYDATA:message}

Maybe using the DATA pattern repeatedly was too expensive?

Details

Assignee

Reporter

Fix versions

Affects versions

Created April 5, 2013 at 7:51 AM
Updated September 21, 2013 at 3:15 PM
Configure