Issues
conditionals do not work on cloned events
Description
discovered while testing
Gliffy Diagrams
Details
Details
Assignee
Reporter
Labels
Fix versions
Affects versions
Activity
Philippe Weber January 28, 2015 at 5:11 AM
Thomas De Smedt October 16, 2013 at 11:17 AMEdited
current workaround: use deprecated 'type' conditions:
filter {
uuid { field => "[uuid]" }
clone { clones => ["cloned"] }
mutate {
type => "initial"
replace => ["message","http://message/%{type}/%{[uuid]}"]
}
}(would need more conditions if it should be usable in pre 1.2 logstash)
Thomas De Smedt October 16, 2013 at 11:05 AMEdited
A quick inspection into the code seems to confirm this: see the config DSL code for filters. It
pushes new events only into the following filters
generates code with conditional checks against the incoming event ('event' variable) instead of the actual 'current' event.
I assume it that logstash needs to choose more clearly (fixed in design or as event property) between:
reintroducing the event as a brand new event at the top of the filter chain (as an internal additional input, but with buffer overflow and infinite loop risks) . I assume this was the behaviour pre logstash1.2.
handling the event downward to the remaining filters (logstash 1.2, but requires fix that conditionals are evaluated against that new event)
sending the event directly to output (with only the type field as a discriminator). Scales best because the buffering is then external with e.g. a file/redis/mq output before you process it again with logstash.
Philippe Weber October 16, 2013 at 9:50 AM
The current problems is that the conditional will always be applied to the original events but the filters will be applied to extra events as well, thus making it difficult to achieve the correct behavior
I tried to clone an event and then modify the original, using conditionals. This does not seem to work.
It seems as if all conditionals work on the original event, instead of the cloned one, although the filter themselves act on the correct event.
input { stdin { type => "initial" } } filter { if [type] == "initial" { uuid { field => "[uuid]" } clone { clones => ["cloned"] } mutate { replace => ["message","http://message/%{[type]}/%{[uuid]}"] } } } output { stdout { codec => json debug => true } }with input
a small messageresults in
{ "message" => "http://message/initial/aadfbd6f-eb32-4c30-8461-68271afc1d3b", "@timestamp" => "2013-10-16T07:53:21.466Z", "@version" => "1", "type" => "initial", "host" => "PO120003533", "uuid" => "aadfbd6f-eb32-4c30-8461-68271afc1d3b" } { "message" => "http://message/cloned/aadfbd6f-eb32-4c30-8461-68271afc1d3b", "@timestamp" => "2013-10-16T07:53:21.466Z", "@version" => "1", "type" => "cloned", "host" => "PO120003533", "uuid" => "aadfbd6f-eb32-4c30-8461-68271afc1d3b" }this while i expect the 'initial' message to say
"message" => "a small message",The config above shows the buggy behaviour, and i tried a number of variations that still don't work
creating a new field with a mutate filter
add_field => ["cloned_type","%{[type]}"]and condition on that. Although the new field is created correctly, the condition still uses the original event.
writing the condition as
"%{[type]}" == "initial"logging of all the steps with a log shows that the cloned event is NOT re-injected at the top of the filter stack (as i assumed). So i expected that the cloned event wouldn't be handled by the
mutateafter theclone. To be sure i also tried a double check:if [type] == "initial" { uuid { field => "[uuid]" } clone { clones => ["cloned"] } if [type] == "initial" { mutate { replace => ["message","http://message/%{type}/%{[uuid]}"] } } }.