1.2.x Conditional + split/clone events is currently broken

Description

Using the conditionals with the filters that emit events is broken and the current behaviors are difficult to understand for end users.

In 1.1.x the newly created events were going through the full filters chain, this was a bug.
in 1.2.x the intended behavior it for them to continue through the filter chain from their creation point.
(As previously discussed in this PullRequest https://github.com/logstash/logstash/pull/555)

The config_ast does not seems robust enough to handle this case accordingly as the observed behaviors differs from expectations.

Current observed behaviors:

  • The split filter is yielding all created event and cancel the original one, this have the impact to pop all the events out of the filterworker chain and they are sent directly to the output.
    It is because it goes through the following code generated by config_ast

  • The clone filter yield the created clones, but keep the original events. Only the original event will be used in if conditionals, but the filters will be applied to all extra_events
    As an example this simple filter config

    generated the following code in config_ast:

Activity

Show:
EdsonF
May 8, 2014, 10:18 PM
Edited

Aaron, I'm forwarding these events to an internal system we have here in the company I work for, and that system accepts only 1 'key=value' by message/request. So, I'm forced to split up multiple metrics/values from a single event into many outputs. It is, indeed, a limitation of that system I'm working with, but it's a possible usage scenario as well in the real world, isn't it?

Aaron Mildenstein
May 8, 2014, 10:20 PM

Indeed! Again, I was just curious.

Have you tried this with 1.4.1 yet?

EdsonF
May 8, 2014, 10:23 PM

No, only with 1.4.0, which was the latest one available by the time. (I posted on this thread just because I couldn't find another one and didn't want to open a duplicate).

Jason Kendall
May 8, 2014, 10:24 PM

1. JIRA-2018 is probably related – or the examples are more clear of the problem.

2. As another example (assuming it is the same as 2018) - You have firewall rules, you want to clone some of the events and send the initial event to your ES backend, but the cloned event you want to toss at the anonymizer and remove some fields for distribution to external parties.

Nathan Young
June 4, 2014, 9:22 PM
Edited

I'm interested in this issue. I have a case where I process emails with CSVs attached.

  1. CSV content goes into message from IMAP input

  2. split{} on "\n" in the message field

  3. run new message field through CSV filter

Currently unable to do step3

Fixed

Assignee

Logstash Developers

Reporter

Philippe Weber

Labels

None

Fix versions

Affects versions

Configure