conditionals do not work on cloned events

Description

I tried to clone an event and then modify the original, using conditionals. This does not seem to work.
It seems as if all conditionals work on the original event, instead of the cloned one, although the filter themselves act on the correct event.

with input

results in

this while i expect the 'initial' message to say

The config above shows the buggy behaviour, and i tried a number of variations that still don't work

  • creating a new field with a mutate filter

    and condition on that. Although the new field is created correctly, the condition still uses the original event.

  • writing the condition as

  • logging of all the steps with a log shows that the cloned event is NOT re-injected at the top of the filter stack (as i assumed). So i expected that the cloned event wouldn't be handled by the mutate after the clone. To be sure i also tried a double check:

    .

Activity

Show:
Philippe Weber
October 16, 2013, 9:50 AM

The current problems is that the conditional will always be applied to the original events but the filters will be applied to extra events as well, thus making it difficult to achieve the correct behavior

Thomas De Smedt
October 16, 2013, 11:05 AM
Edited

A quick inspection into the code seems to confirm this: see the config DSL code for filters. It

  • pushes new events only into the following filters

  • generates code with conditional checks against the incoming event ('event' variable) instead of the actual 'current' event.

I assume it that logstash needs to choose more clearly (fixed in design or as event property) between:

  • reintroducing the event as a brand new event at the top of the filter chain (as an internal additional input, but with buffer overflow and infinite loop risks) . I assume this was the behaviour pre logstash1.2.

  • handling the event downward to the remaining filters (logstash 1.2, but requires fix that conditionals are evaluated against that new event)

  • sending the event directly to output (with only the type field as a discriminator). Scales best because the buffering is then external with e.g. a file/redis/mq output before you process it again with logstash.

Thomas De Smedt
October 16, 2013, 11:17 AM
Edited

current workaround: use deprecated 'type' conditions:

(would need more conditions if it should be usable in pre 1.2 logstash)

Philippe Weber
January 28, 2015, 5:11 AM

Assignee

Colin Surprenant

Reporter

Thomas De Smedt

Fix versions

Affects versions

Configure