Metric filter events (possibly other filters that add events to the stream) are not applied against conditionals correctly
Description
discovered while testing
testing discovered
Gliffy Diagrams
Activity
Lex Neva January 26, 2015 at 6:59 PM
Sure will. That's some epic work there
Philippe Weber January 24, 2015 at 11:58 AM
Fixed in https://github.com/elasticsearch/logstash/pull/1545
This PR will become famous
Ryan O'Keeffe April 23, 2014 at 8:06 PM
https://github.com/elasticsearch/logstash/pull/1260 will fix this. Just needs spec tests and a pull into master. So close!
Daniel Dreier April 19, 2014 at 7:30 AM
I'm still seeing this behavior in 1.4.0. If I have put a metrics filter before a grok filter, the grok filter (which is behind a conditional) tramples all over the data in the metric event.
Ryan O'Keeffe January 21, 2014 at 9:33 PM
I agree that the most logical solution would be what Jordan outlined as essentially the resulting events would continue in the pipeline where they were generated. Implementing this is probably complex, but ultimately I would say its the best solution. That said, if its easier in the meantime to push a patch out that flushes events to the beginning and is documented, the community might benefit from being able to use functionality until a final comprehensive fix can be completed.
EDIT: The metrics filter injects its events into the stream, but those events are not applied against conditionals in the intended manner.
Gist from electrical illustrating this:
input { generator { } } filter { metrics { meter => "events" add_tag => metrics } } filter { if [type] == "bla" { noop { add_field => [ "blaat", "grrr" ] } } } output { if "metrics" in [tags] { stdout { codec => rubydebug } } } [richard@devel001 testing]$ java -jar logstash-1.3.2-flatjar.jar agent -f ls_metrics_ml.conf Using milestone 1 filter plugin 'metrics'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin. For more information on plugin milestones, see http://logstash.net/docs/1.3.2/plugin-milestones {:level=>:warn} Using milestone 2 filter plugin 'noop'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.3.2/plugin-milestones {:level=>:warn} { "@version" => "1", "@timestamp" => "2013-12-30T17:15:41.552Z", "message" => "devel001.misc.de01.ispavailability.com", "events.count" => 5406, "events.rate_1m" => 1083.0, "events.rate_5m" => 1083.0, "events.rate_15m" => 1083.0, "tags" => [ [0] "metrics" ], "blaat" => "grrr" }Original find:
The metrics filter does not seem to work if there is also a multiline filter in the configuration. It doesn't matter if the events match the multiline filter or not. If the multiline filter is commented out, metrics work as expected. No errors or messages are logged. Here is a very simplified config to illustrate this:
input { generator { } } filter { metrics { meter => "events" add_tag => metrics } } filter { if [type] == "testmultiline" { multiline { pattern => "^<\d\d\d\d-\d\d-\d\d" what => "previous" negate => "true" } } } output { if "metrics" in [tags] { stdout { codec => rubydebug } } }