in/not in operator and single element lists
Description
Gliffy Diagrams
Activity
Philippe Weber January 27, 2015 at 5:06 AM
Closing as duplicate in favor of github issues, please continue the discussion and follow-up there.
https://github.com/elasticsearch/logstash/issues/1361
https://github.com/elasticsearch/logstash/issues/1850
https://github.com/elasticsearch/logstash/issues/2129
Matt Dainty May 12, 2014 at 2:01 PM
I realised the --debug flag spits out the expanded grammar, and now I can see why it doesn't work:
...
@logger.debug? && @logger.debug("filter received", :event => event.to_hash)
if (((x = event["['bar']"]; x.respond_to?(:include?) && x.include?(event["[foo]"]))))
newevents = []
...So it looks my hunch was right, it's trying to look up a field called literally "'bar'" (as opposed to just "bar"). With that I worked out where the problem was in the grammar definition and I've created PR #1361 which fixes it.
Philippe Weber May 12, 2014 at 1:24 PM
You have indeed spotted a bug where the config parser get confused and interpret the single-value array as a field reference.
For information you can see the generated pipeline using the --debug flag and see that event("bar") is called instead of using a static array.
if (((x = event["[\"bar\"]"]; x.respond_to?(:include?) && x.include?(event["[foo]"]))))As a workaround, adding a dummy extra-value to the array should give you the expected result
{
"message" => "hello\r",
"@version" => "1",
"@timestamp" => "2014-05-12T13:19:57.211Z",
"foo" => "bar",
"tags" => [
[0] "[foo] in ['bar'] correct",
[1] "[foo] not in ['bar'] correct"
]
}
Assuming the following test configuration, neither of the two in/not in conditionals work as expected when supplied with a list of values containing only a single (matching) element:
input { generator { count => 1 add_field => { foo => bar } } filter { if [foo] in ['bar'] { noop { add_tag => "[foo] in ['bar'] correct" } } else { noop { add_tag => "BUG! [foo] in ['bar'] failed!" } } if [foo] not in ['bar'] { noop { add_tag => "BUG! [foo] not in ['bar'] failed!" } } else { noop { add_tag => "[foo] not in ['bar'] correct" } } } output { stdout { codec => rubydebug } }I get:
{ "message" => "Hello world!", "@version" => "1", "@timestamp" => "2014-05-12T11:05:43.905Z", "foo" => "bar", "host" => "localhost", "sequence" => 0, "tags" => [ [0] "BUG! [foo] in ['bar'] failed!", [1] "BUG! [foo] not in ['bar'] failed!" ] }Either removing the "[]"'s to flatten the list into a string or adding additional values to make the list contain more than one element fixes the conditionals and they work as expected.
I've tried poking about with the internals under lib/logstash/config/* however I'm not entirely sure where/what the bug is. My current hunch is the parser is perhaps getting confused between list and field syntax?
Either way, the parser accepts the syntax but it doesn't work as expected.