drop does not work with exclude_tags
Description
Attachments
testing discovered
Gliffy Diagrams
Activity
Philippe Weber April 19, 2014 at 12:21 PM
Jason, I don't get what you mean by "other than events"... everything is event in logstash.
The root cause is that the filter execution flow is broken since 1.2.0 for filter that "emits events" metrics,split,clone
Focusing on the drop filter, its purpose is that with using conditionals outside the filter it was a replacement of the grep filter, used a lot before 1.2.x. Plus all the "in" logic used (tags,exclude_tag,type) are now deprecated in favor of conditional.
That's mostly why drop filter should not have any logic.
However due to the bug discussed here, it is a good temporary workaround.
The timeline I suppose would be:
1. apply the patch discussed here for 1.4.1 to unblock such use case for users struggling with it since 1.2.0
2. have https://github.com/elasticsearch/logstash/pull/1260 fixing this long running issue in 1.5.x
3. remove deprecated "in" logic in 1.5.x thus deprecating the filter/base filter? method that is called by each
4. change drop filter to not inherit from filter/base and thus having ZERO configuration because add_field,add_tag has really no meaning for a dropped event.
I would happily propose a pull request for point 4. in the future if you agree with the action plan
Jason Kendall April 18, 2014 at 3:18 PM
Didn't say it wasn't valid.
Correct me if I'm wrong (which is very possible) the patch wouldn't allow you to drop anything other then events. Not sure if there's a use case to drop anything else.
Philippe Weber April 18, 2014 at 4:44 AM
The patch is valid! at least as a workaround.
The complex issue comes from a great bug in the pipeline as identified with clone filter that is also emitting events without cancelling the original
If you run logstash with the -vv flag you see the pipeline code, where the if !("metrics in [tags]) is always evaluated against the original event, so all your metrics events goes to the drop filter too
Jason Kendall April 17, 2014 at 7:49 PM
Removing exclude_tags (which is deprecated) yields correct results.
What I don't understand, since the IF fails why is it still hitting the drop. Not sure the patch is what we needed.
Markus Linnala March 21, 2014 at 3:10 PM
With patch this works as expected. But when I drop exclude_tags it does not work. That is same as
I'd expect to get metrics output every five seconds. But do not get any.
drop has double guard for metric events but they are not enough not to drop event.
Nothing to see here really.