Mutate: cannot remove a deep field

Description

When using mutate add_field directive with nested fields and nested field is not present: this does not result in the non-presence of the field to add but rather results in redundant data.
Additionally it is not possible to mutate remove a nested field.
In the attachment a config file and the used JSON in/out.
Although I am not sure whether there exists a workaround for these cases.

Attachments

2

Gliffy Diagrams

Activity

Show:

Jordan Sissel August 16, 2013 at 6:37 AM

This is fixed and will be available in the next release.

AmitS August 16, 2013 at 5:54 AM

I came across a similar use case where I need to conditionally apply a mutate filter based on custom field values. I realized that grep or grok filters do not allow "matching" on custom fields.

i.e. If I apply a grep filter
grok { #grok filter for reference
pattern => "(?<log_timestamp>%{MONTHNUM}%{MONTHDAY} %{TIME})(.

.

|\s*%{LOGLEVEL:log_level}\s*[(?<thread_name>.*?)]%{GREEDYDATA:message_remainder})?"
}

grep {
match => [ "@message_remainder", ".+" ] #Verify if the field has value
add_tag => [ "parsing_succeeded" ]
}

mutate {
tags => [ "parsing_succeeded" ]
replace => [ "@message", "%{message_remainder}" ]
}

@message is never replaced by the value of %{message_reminder} which indicates grep didn't succeed for custom fields. Is this issue tracking this same issue?

dsfsd March 6, 2013 at 8:24 AM

@Philippe: thx for the info. I now have a working config!

For back reference in relation to the grep-filter:

  • In version 1.1.9 it doesn't matter if you try with or without the "@fields." -> fails to find the field and add_field is not performed

  • In the last dev build of version 1.1.10: the "@fields." doesn't work, without "@fields." it works

  • In the nightly build version 1.1.10.dev: the "@fields." doesn't work, without "@fields." it works

Philippe Weber March 6, 2013 at 6:32 AM
Edited

@fields is almost "internal" to logstash, don't use it as a prefix in your config, and it will work.
Copy-paste of my proposed code, should have did it

Only remaining difference, is that I'm using custom build that is after 1.1.9, so maybe you would have to try last dev build
https://logstash.objects.dreamhost.com/builds/logstash-1.1.10.dev-monolithic.jar

or even a nightly
http://r.logstash.net/jenkins/job/logstash.jar.daily/lastSuccessfulBuild/artifact/build/logstash-1.1.10.dev-monolithic.jar

dsfsd March 6, 2013 at 6:19 AM
Edited

I have to get back on your workaround with the grep filter.

When I use this conf filter section no "token" field is added:

In the logstash logs I see, surely the "field not present" isn't correct:

Fixed

Details

Assignee

Reporter

Fix versions

Affects versions

Created March 5, 2013 at 7:52 PM
Updated August 16, 2013 at 6:37 AM
Resolved August 16, 2013 at 6:37 AM