not able to use anonymize filter to anonymize a field created by the mutate filter
Description
Gliffy Diagrams
Activity
Show:

Vimil Saju March 16, 2013 at 7:25 AM
@luoxinwei1 I think you should use a separator when joining arrays otherwise ['a','bc'] and ['ab','c'] will get the same hash

luoxinwei1 March 14, 2013 at 8:48 AM
My solution is:
anonymize.rb
here is the config i have
input {
tcp {
host => "xxx.xx.xx.com"
port => 5003
type => "log4j"
}
}
filter {
multiline {
type => "log4j"
pattern => "^((?!(SID.)*$"
what => "previous"
}
}
filter {
grep {
type => "log4j"
match => ["@message", ".Exception."]
}
}
filter {
grok {
type => "log4j"
pattern => "%{TIMESTAMP_ISO8601:timestamp} [(?<env>[a-zA-Z0-9_])] [(?<app>[a-zA-Z0-9_])] (?<logtype>[a-zA-Z0-9_]+) (SID: (?<session_id>[0-9A-F]))(?<errorMessage>.)(?<stackTrace>(?m).*)"
}
}
filter {
mutate {
type => "log4j"
add_field => [ "stackTraceHash", "%{@fields.stackTrace}" ]
}
}
filter {
anonymize {
type => "log4j"
algorithm => "MD5"
fields => ['stackTraceHash']
key =>"longencryptionkey"
}
}
output {
elasticsearch {
host => "127.0.0.1"
port => 9300
}
}
I can't get the field stackTraceHash anonymized
This is the error I see in logstash.log
#<NotImplementedError: (TypeError) can't convert Array into String>, :backtrace=>["org/jruby/ext/openssl/HMAC.java:99:in `hexdigest'", "file:/D:/tools/logstash/logstash-1.1.9-monolithic.jar!/logstash/filters/anonymize.rb:51:in `anonymize_openssl'", "file:/D:/tools/logstash/logstash-1.1.9-monolithic.jar!/logstash/filters/anonymize.rb:40:in `filter'", "org/jruby/RubyArray.java:1612:in `each'", "file:/D:/tools/logstash/logstash-1.1.9-monolithic.jar!/logstash/filters/anonymize.rb:39:in `filter'", "file:/D:/tools/logstash/logstash-1.1.9-monolithic.jar!/logstash/filters/base.rb:89:in `execute'", "file:/D:/tools/logstash/logstash-1.1.9-monolithic.jar!/logstash/filterworker.rb:87:in `filter'", "org/jruby/RubyArray.java:1612:in `each'", "file:/D:/tools/logstash/logstash-1.1.9-monolithic.jar!/logstash/filterworker.rb:77:in `filter'", "org/jruby/RubyArray.java:1612:in `each'", "file:/D:/tools/logstash/logstash-1.1.9-monolithic.jar!/logstash/filterworker.rb:76:in `filter'", "file:/D:/tools/logstash/logstash-1.1.9-monolithic.jar!/logstash/filterworker.rb:38:in `run'", "file:/D:/tools/logstash/logstash-1.1.9-monolithic.jar!/logstash/agent.rb:738:in `run_filter'", "file:/D:/tools/logstash/logstash-1.1.9-monolithic.jar!/logstash/agent.rb:452:in `run_with_config'"]