Mutate: cannot remove a deep field

This is fixed and will be available in the next release.

I came across a similar use case where I need to conditionally apply a mutate filter based on custom field values. I realized that grep or grok filters do not allow "matching" on custom fields.

i.e. If I apply a grep filter
grok { #grok filter for reference
pattern => "(?<log_timestamp>%{MONTHNUM}%{MONTHDAY} %{TIME})(.

|\s*%{LOGLEVEL:log_level}\s*[(?<thread_name>.*?)]%{GREEDYDATA:message_remainder})?"
}

grep {
match => [ "@message_remainder", ".+" ] #Verify if the field has value
add_tag => [ "parsing_succeeded" ]
}

mutate {
tags => [ "parsing_succeeded" ]
replace => [ "@message", "%{message_remainder}" ]
}

@message is never replaced by the value of %{message_reminder} which indicates grep didn't succeed for custom fields. Is this issue tracking this same issue?

@Philippe: thx for the info. I now have a working config!

For back reference in relation to the grep-filter:

• In version 1.1.9 it doesn't matter if you try with or without the "@fields." -> fails to find the field and add_field is not performed

• In the last dev build of version 1.1.10: the "@fields." doesn't work, without "@fields." it works

• In the nightly build version 1.1.10.dev: the "@fields." doesn't work, without "@fields." it works

@fields is almost "internal" to logstash, don't use it as a prefix in your config, and it will work.
Copy-paste of my proposed code, should have did it

Only remaining difference, is that I'm using custom build that is after 1.1.9, so maybe you would have to try last dev build
https://logstash.objects.dreamhost.com/builds/logstash-1.1.10.dev-monolithic.jar

or even a nightly
http://r.logstash.net/jenkins/job/logstash.jar.daily/lastSuccessfulBuild/artifact/build/logstash-1.1.10.dev-monolithic.jar

I have to get back on your workaround with the grep filter.

When I use this conf filter section no "token" field is added:

In the logstash logs I see, surely the "field not present" isn't correct: