Logstash stops filtering
Description
Attachments
2
Gliffy Diagrams
Activity
Show:
Philippe Weber August 1, 2014 at 12:00 PM
A little more aggressive garbage-collect on jira.com, expecting follow-up activities to take place on github issues
Jason Kendall March 5, 2014 at 4:11 PM
Is the filtering issue still happening in 1.3.3?
The questions are best served in the mailing list, or IRC.
Hi,
I will explain the problem I am facing first, then ask a couple of questions on how to tweak logstash.
1)The Problem:
Logstash filters are not working after a while. Logstash only understands the @type, but none of the filters are working.
Output of kibana showing both working and a broken one is in the attachment.
I believe logstash restarts and it does not work, because previously I was using an experimental plugin and experimental warning was in the outputs.
I blamed the experimental plugin for this bug, but it still happens after removing it.
----------------------------------------------
2) When I check the memory usage I see that logstash is using 600 MB.
a) Could this be the issue? Would starting logstash with Java's -Xms and -Xmx options work ?
b) I am working on 2 different types of documents. Logcat and apache logs. Would it be better if I split them in different config files and start 2 different logstash instances?
c) Logstash working with a single accumulating log file is a well tested usage. (like apache access logs)
With android logcat logs, they come as a new document, and does not append to the previous one, hence there would be too many documents. Should I use a diffrent strategy, like deleting old files after a while, so that logstash does not have to listen changes on those documents?
Thanks in advance.
Ali