Grok Discrepancies in Logstash 1.1.1 but works in Logstash 1.1.0

Description

I have recently switched to Logstash 1.1.1 but some how there are very big discrepancies in indexing from grok. Somehow I am getting _grokparsefailure and the logs that are indexed appear different from the standard output of the logstash instance.

This is the config I am using:

These are the logs that was indexed into elasticsearch via logstash, and somehow many of the logs seemed missing:

But these are the standard logs that is going through the file:

The settings worked for logstash 1.1.0 previously, so I am not sure what happened. What can be done to rectify the situation?

Activity

Show:
Philippe Weber
August 1, 2012, 9:10 AM

Could you please wrap your configuration using {code}{code} tags, to ease reading for others.

By comparing your two grok filters, it seems to me that you fall in the same case as I described in
because the {INT:cache_bool} would also catch the "-" that you expect in your second filter but it will fail as no other digits follow, thus causing watchdog error.

If I misunderstood your problem can you please try to pin-point the failing input lines and your expected output by using stdin/sdtout input/output in your configuration.

As a workaround for your use case it seems sufficient to use POSINT (or maybe NONNEGINT if 0 is valid)
Or you could put the two patterns definition inside the same grok filter but re-order them as having break_on_match=true (default value) will avoid falling in the error state I described above where INT regex tries to match "-".

Jonathan Moo
August 2, 2012, 2:51 AM
Edited

Updated it with code tags.

Actually for the cache_bool, my logs either pushes booleans or dashes if none; but I wanted the dashes to be 0 instead. Can it be done using grok? If it can be done and if it also solves the watchdog error, then it will solve my problem.

Richard Pijnenburg
December 18, 2012, 1:59 AM

Cleanup of old/stalled ticket.
If issue still persists with current version please open a new ticket.

Assignee

Logstash Developers

Reporter

Jonathan Moo

Fix versions

Affects versions

Configure