logstash file input looses a lot of lines
Description
Gliffy Diagrams
Activity
Show:
Details
Details
Assignee
Logstash Developers
Logstash DevelopersReporter
Klavs Klavsen
Klavs KlavsenLabels
Affects versions
Created April 9, 2014 at 1:17 PM
Updated February 13, 2015 at 10:06 PM
I just used logstash on a file input - where we simply throw in finished files (they get uploaded from the router, at set intervals), and then we remove them after a couple of hours (to avoid running out of diskspace).
Problem was that with input { file { path => "/path/*", start_position => beginning } -it lost a lot of lines at set intervals, which seemed to fit with the size/time-window for each logfile.
It should be very easy to reproduce (throw a lot of lines in logfiles at it, and then compare counts to actual content in logfiles).
I had to get it working, so I figured I'd try something else to check for logfiles, and switched to using logstash-forwarder (aka. lumberjack) to handle it, and then just have a lumberjack input in logstash - and that works flawlessly.