Filter plugins applied to the input after the Split plugin are skipped on Logstash 1.2.x

Description

Starting with the version 1.2.x it seems that a split filter followed by some grok pattern matching does not work anymore. No matter what you try, after the split filtering the grok patterns are never applied to the splitted log lines. It's like the entire grok section is completely skipped by Logstash after the split filter.

This happens to other filters as well: after the split filter, every other filter plugin is skipped (tried: mutate, metaevent, etc.).

Note: to make the test case more simple to reproduce I am just using the exec plugin, with a simple cat on a text file. In a real setup the logs are extracted from the body of an email, using the imap plugin.

Here is an example of the config file I have used with Logstash 1.2.x:

The log lines are structured with the following format:

daemon:hostname:command: some random message

The pattern I have used is really simple:

Using the "tag_on_failure" option with Grok does not seem to have any effect either.

On the other hand, a similar configuration used with Logstash 1.1.13 works perfectly.

Activity

Show:
Philippe Weber
September 19, 2013, 9:52 AM

Are you sure your email line termination is only "\n" and not "\r\n".
If possible, could you please attach your raw email content for me to try to reproduce

Matteo Dessalvi
September 19, 2013, 1:55 PM
Edited

I am pretty sure there are no "\r" characters in the email body, only "\n" at the end of the strings.

I was probaly not so clear in my explanation: the split filter is working perfectly by itself, which means that it is acting
as described in the docs, splitting the email body in multiple events using the "\n" character as a default terminator.
After passing through the split filter the "splitted" tag is attached as well, which means that the filter has done its job
correctly.

But the problem arise once you need to pass all these events to grok (or some other filter): they are just
completely bypassed: you can see that because the grok filter will not add any tag in case of failure (option:
tag_on_failure).

If you try the example config I have posted you'll see that this behavior is showing up even with just using a
'/bin/cat textfile', so I do believe it 's not a problem of formatting the email body.

On the other hand, if the split plugin is not involved, which mean I am analyzing an event with only one line
of text then the grok filter will work and happily apply the pattern. It looks like that the execution of split and grok
in the same filter section is mutually exclusive.

As I also stated above a similar configuration is working correctly with the 1.1.13 version of Logstash.

Matteo Dessalvi
December 10, 2013, 1:03 PM

Just a small update: applying the patch proposed by the following pull request: https://github.com/logstash/logstash/pull/793
the problem I have described was solved.

It would be great if this PR would be taken into consideration as a bug fix and introduced in some of the future stable branches
of Logstash.

Jason Kendall
April 28, 2014, 6:40 PM

Confirmed 1.4.0 – PR - Assigning.

Philippe Weber
January 28, 2015, 5:17 AM

Assignee

Colin Surprenant

Reporter

Matteo Dessalvi

Labels

Fix versions

Affects versions

Configure