Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device. Atlassian cookies and tracking notice, (opens new window)
I just used logstash on a file input - where we simply throw in finished files (they get uploaded from the router, at set intervals), and then we remove them after a couple of hours (to avoid running out of diskspace).
Problem was that with input { file { path => "/path/*", start_position => beginning } -it lost a lot of lines at set intervals, which seemed to fit with the size/time-window for each logfile.
It should be very easy to reproduce (throw a lot of lines in logfiles at it, and then compare counts to actual content in logfiles).
I had to get it working, so I figured I'd try something else to check for logfiles, and switched to using logstash-forwarder (aka. lumberjack) to handle it, and then just have a lumberjack input in logstash - and that works flawlessly.
I just used logstash on a file input - where we simply throw in finished files (they get uploaded from the router, at set intervals), and then we remove them after a couple of hours (to avoid running out of diskspace).
Problem was that with input { file { path => "/path/*", start_position => beginning } -it lost a lot of lines at set intervals, which seemed to fit with the size/time-window for each logfile.
It should be very easy to reproduce (throw a lot of lines in logfiles at it, and then compare counts to actual content in logfiles).
I had to get it working, so I figured I'd try something else to check for logfiles, and switched to using logstash-forwarder (aka. lumberjack) to handle it, and then just have a lumberjack input in logstash - and that works flawlessly.