Mutate: cannot remove a deep field
Description
Attachments
Gliffy Diagrams
Activity
Jordan Sissel August 16, 2013 at 6:37 AM
This is fixed and will be available in the next release.

AmitS August 16, 2013 at 5:54 AM
I came across a similar use case where I need to conditionally apply a mutate filter based on custom field values. I realized that grep or grok filters do not allow "matching" on custom fields.
i.e. If I apply a grep filter
grok { #grok filter for reference
pattern => "(?<log_timestamp>%{MONTHNUM}%{MONTHDAY} %{TIME})(.
. |
|\s*%{LOGLEVEL:log_level}\s*[(?<thread_name>.*?)]%{GREEDYDATA:message_remainder})?"
}
grep {
match => [ "@message_remainder", ".+" ] #Verify if the field has value
add_tag => [ "parsing_succeeded" ]
}
mutate {
tags => [ "parsing_succeeded" ]
replace => [ "@message", "%{message_remainder}" ]
}
@message is never replaced by the value of %{message_reminder} which indicates grep didn't succeed for custom fields. Is this issue tracking this same issue?

dsfsd March 6, 2013 at 8:24 AM
@Philippe: thx for the info. I now have a working config!
For back reference in relation to the grep-filter:
In version 1.1.9 it doesn't matter if you try with or without the "@fields." -> fails to find the field and add_field is not performed
In the last dev build of version 1.1.10: the "@fields." doesn't work, without "@fields." it works
In the nightly build version 1.1.10.dev: the "@fields." doesn't work, without "@fields." it works

Philippe Weber March 6, 2013 at 6:32 AMEdited
@fields is almost "internal" to logstash, don't use it as a prefix in your config, and it will work.
Copy-paste of my proposed code, should have did it
Only remaining difference, is that I'm using custom build that is after 1.1.9, so maybe you would have to try last dev build
https://logstash.objects.dreamhost.com/builds/logstash-1.1.10.dev-monolithic.jar
or even a nightly
http://r.logstash.net/jenkins/job/logstash.jar.daily/lastSuccessfulBuild/artifact/build/logstash-1.1.10.dev-monolithic.jar

dsfsd March 6, 2013 at 6:19 AMEdited
I have to get back on your workaround with the grep filter.
When I use this conf filter section no "token" field is added:
In the logstash logs I see, surely the "field not present" isn't correct:
Details
Details
Assignee
Reporter

When using mutate add_field directive with nested fields and nested field is not present: this does not result in the non-presence of the field to add but rather results in redundant data.
Additionally it is not possible to mutate remove a nested field.
In the attachment a config file and the used JSON in/out.
Although I am not sure whether there exists a workaround for these cases.