Several times a day, I'll suddenly encounter big gaps in my log data. Each time, the logstash indexer process is doing nothing but looping while spitting this into the logstash logs:
1
{:timestamp=>"2013-11-03T02:03:05.644000+0000", :message=>"Failed to flush outgoing items", :outgoing_count=>33, :exception=>#<JSON::GeneratorError: source sequence is illegal/malformed utf-8>, :backtrace=>["json/ext/GeneratorMethods.java:71:in `to_json'", "file:/opt/logstash/logstash.jar!/logstash/event.rb:169:in `to_json'", "file:/opt/logstash/logstash.jar!/logstash/outputs/elasticsearch.rb:163:in `flush'", "org/jruby/RubyArray.java:1617:in `each'", "file:/opt/logstash/logstash.jar!/logstash/outputs/elasticsearch.rb:158:in `flush'", "file:/opt/logstash/logstash.jar!/stud/buffer.rb:219:in `buffer_flush'", "org/jruby/RubyHash.java:1332:in `each'", "file:/opt/logstash/logstash.jar!/stud/buffer.rb:216:in `buffer_flush'", "file:/opt/logstash/logstash.jar!/stud/buffer.rb:193:in `buffer_flush'", "file:/opt/logstash/logstash.jar!/stud/buffer.rb:112:in `buffer_initialize'", "org/jruby/RubyKernel.java:1489:in `loop'", "file:/opt/logstash/logstash.jar!/stud/buffer.rb:110:in `buffer_initialize'"], :level=>:warn}Each time, the only thing that seems to get it working again is to restart the indexer process. Unfortunately, I never do get back the lost log events in logstash.
I am sending in data from international customers, which frequently contains Norwegian characters, Chinese characters, and Japanese characters.
I'd expect a logging system to at worst drop the single offending line with a warning tag, and process the rest. I don't however, expect it to have one problematic line cause the whole thing to stop processing while it's hung on a bad entry.
The indexer is logstash 1.2.2 on Ubuntu 12.04, the source events are from log4net sent to logstash across UDP via the log4net UdpAppender.