Several times a day, I'll suddenly encounter big gaps in my log data. Each time, the logstash indexer process is doing nothing but looping while spitting this into the logstash logs:
Each time, the only thing that seems to get it working again is to restart the indexer process. Unfortunately, I never do get back the lost log events in logstash.
I am sending in data from international customers, which frequently contains Norwegian characters, Chinese characters, and Japanese characters.
I'd expect a logging system to at worst drop the single offending line with a warning tag, and process the rest. I don't however, expect it to have one problematic line cause the whole thing to stop processing while it's hung on a bad entry.
The indexer is logstash 1.2.2 on Ubuntu 12.04, the source events are from log4net sent to logstash across UDP via the log4net UdpAppender.