Do not update sincedb if the data is not accepted by elasticsearch


I am using elasticsearch-1.1.1 with logstash 1.4.0 and I am currently in process of parsing 1.2B documents across 18 indices.
At times, due to various reasons (probably a bug in LS), I see these errors:

The problem vanishes after I restart the logstash process.

Anyway, the major problem is that even though the write has failed, logstash updates sincedb and doesn't retry the loglines, this has caused a large pain as one has to constantly monitor and restart if such errors are found. So far I have lost over 100m documents and no idea how to get them back !


Logstash Developers



Affects versions