Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device. Atlassian cookies and tracking notice, (opens new window)
Logstash bundes a custom version of the filewatch gem, which is buggy. Its reads data in 16kB chunks, and tries to sync the sincedb after each chunk. However, this is flawed, because 1) mid-line file offsets are being saved 2) when all chunks including the last one are processed in less than 10s, the last position (eof) is not saved. When logstash is killed and restarted, duplicate entries are emitted.
Logstash bundes a custom version of the filewatch gem, which is buggy. Its reads data in 16kB chunks, and tries to sync the sincedb after each chunk. However, this is flawed, because 1) mid-line file offsets are being saved 2) when all chunks including the last one are processed in less than 10s, the last position (eof) is not saved. When logstash is killed and restarted, duplicate entries are emitted.
sincedb should not be updated in the read loop, but only at the end. https://github.com/jordansissel/ruby-filewatch has the correct code.