We're updating the issue view to help you get more done. 

logstash indexer gets stuck frequently attempting to flush outgoing to elastisearch

Description

Several times a day, I'll suddenly encounter big gaps in my log data. Each time, the logstash indexer process is doing nothing but looping while spitting this into the logstash logs:

1 {:timestamp=>"2013-11-03T02:03:05.644000+0000", :message=>"Failed to flush outgoing items", :outgoing_count=>33, :exception=>#<JSON::GeneratorError: source sequence is illegal/malformed utf-8>, :backtrace=>["json/ext/GeneratorMethods.java:71:in `to_json'", "file:/opt/logstash/logstash.jar!/logstash/event.rb:169:in `to_json'", "file:/opt/logstash/logstash.jar!/logstash/outputs/elasticsearch.rb:163:in `flush'", "org/jruby/RubyArray.java:1617:in `each'", "file:/opt/logstash/logstash.jar!/logstash/outputs/elasticsearch.rb:158:in `flush'", "file:/opt/logstash/logstash.jar!/stud/buffer.rb:219:in `buffer_flush'", "org/jruby/RubyHash.java:1332:in `each'", "file:/opt/logstash/logstash.jar!/stud/buffer.rb:216:in `buffer_flush'", "file:/opt/logstash/logstash.jar!/stud/buffer.rb:193:in `buffer_flush'", "file:/opt/logstash/logstash.jar!/stud/buffer.rb:112:in `buffer_initialize'", "org/jruby/RubyKernel.java:1489:in `loop'", "file:/opt/logstash/logstash.jar!/stud/buffer.rb:110:in `buffer_initialize'"], :level=>:warn}

Each time, the only thing that seems to get it working again is to restart the indexer process. Unfortunately, I never do get back the lost log events in logstash.

I am sending in data from international customers, which frequently contains Norwegian characters, Chinese characters, and Japanese characters.

I'd expect a logging system to at worst drop the single offending line with a warning tag, and process the rest. I don't however, expect it to have one problematic line cause the whole thing to stop processing while it's hung on a bad entry.

The indexer is logstash 1.2.2 on Ubuntu 12.04, the source events are from log4net sent to logstash across UDP via the log4net UdpAppender.

Environment

None

Status

Assignee

Logstash Developers

Reporter

Aaron Brown

Affects versions

1.2.2

Priority