Unexpected exception using logstash xml filter

Description

I'm having trouble using Logstash to process XML log files.

I've followed this documentation http://logstash.net/docs/1.2.2/filters/xml but no matter how I configure the tool I get the following exception after which logstash appears to stop processing events.

Exception in filterworker {"exception"=>#<NoMethodError: undefined method `[]' for nil:NilClass>, "backtrace"=>["file:/home/xDownloads/logstash/logstash-1.2.2-flatjar.jar!/logstash/event.rb:113:in `[]'", "file:/home/x/Downloads/logstash/logstash-1.2.2-flatjar.jar!/logstash/event.rb:170:in `include?'", "file:/home/x/Downloads/logstash/logstash-1.2.2-flatjar.jar!/logstash/filters/xml.rb:78:in `filter'", "(eval):24:in `initialize'", "org/jruby/RubyProc.java:271:in `call'", "file:/home/x/Downloads/logstash/logstash-1.2.2-flatjar.jar!/logstash/pipeline.rb:250:in `filter'", "file:/home/x/Downloads/logstash/logstash-1.2.2-flatjar.jar!/logstash/pipeline.rb:191:in `filterworker'", "file:/home/x/Downloads/logstash/logstash-1.2.2-flatjar.jar!/logstash/pipeline.rb:134:in `start_filters'"], :level=>:error}

My shipper configuration looks like this:

input {
stdin { debug=>"true" type => "xml_trace" }
}
filter {
if [type] == "xml_trace" {
xml {
xpath => [ "/log/event/text()", "m" ]
}
}
}
output {
stdout { codec => rubydebug }
}

When I run my tests I see the following:
[x@xx logstash]$ java -version
java version "1.7.0"
Java(TM) SE Runtime Environment (build pxa6470sr4fp2-20130426_01(SR4 FP2))
IBM J9 VM (build 2.6, JRE 1.7.0 Linux amd64-64 Compressed References 20130422_146026 (JIT enabled, AOT enabled)
J9VM - R26_Java726_SR4_FP2_20130422_1320_B146026
JIT - r11.b03_20130131_32403ifx4
GC - R26_Java726_SR4_FP2_20130422_1320_B146026_CMPRSS
J9CL - 20130422_146026)
JCL - 20130425_01 based on Oracle 7u21-b09
[x@xx logstash]$
[x@xx logstash]$ java -jar logstash-1.2.2-flatjar.jar agent -f ./shipper.conf
Using milestone 1 filter plugin 'xml'. This plugin should work, but would benefit from use by folks like you. Please let us know if you find bugs or have suggestions on how to improve this plugin. For more information on plugin milestones, see http://logstash.net/docs/1.2.2/plugin-milestones {:level=>:warn}
<log><event>text</event></log>
Exception in filterworker {"exception"=>#<NoMethodError: undefined method `[]' for nil:NilClass>, "backtrace"=>["file:/home/x/Downloads/logstash/logstash-1.2.2-flatjar.jar!/logstash/event.rb:113:in `[]'", "file:/home/x/Downloads/logstash/logstash-1.2.2-flatjar.jar!/logstash/event.rb:170:in `include?'", "file:/home/x/Downloads/logstash/logstash-1.2.2-flatjar.jar!/logstash/filters/xml.rb:78:in `filter'", "(eval):24:in `initialize'", "org/jruby/RubyProc.java:271:in `call'", "file:/home/x/Downloads/logstash/logstash-1.2.2-flatjar.jar!/logstash/pipeline.rb:250:in `filter'", "file:/home/x/Downloads/logstash/logstash-1.2.2-flatjar.jar!/logstash/pipeline.rb:191:in `filterworker'", "file:/home/x/Downloads/logstash/logstash-1.2.2-flatjar.jar!/logstash/pipeline.rb:134:in `start_filters'"], :level=>:error}
^CInterrupt received. Shutting down the pipeline. {:level=>:warn}

[x@xx logstash]$

I've noticed that Logstash 1.2.2 uses Nokogiri for dealing with XML. I're read that JRuby can sometimes have problems with Nokogiri (http://jira.codehaus.org/browse/JRUBY-4555) though I'm not sure that is relevant for my issue.

Does anyone have any ideas or suggestions?

Activity

Show:
Jorg Heymans
February 10, 2014, 1:40 PM

Add either
store_xml => false
or

target => "doc"

Josh
February 28, 2014, 2:26 PM

Having this same issue - on 1.3.3 though. Tried in 1.2.2 and had the same issues though. Bump if anyone can help..

Philippe Weber
April 11, 2014, 8:15 AM

It is because you need to specify the source field, even if it is not marked as required

Philippe Weber
April 11, 2014, 8:15 AM
Philippe Weber
August 4, 2014, 10:21 AM

Closing as duplicate of the PR

Assignee

Philippe Weber

Reporter

Greg Lanthier

Labels

Affects versions

Configure