Logstash stops filtering

Description

Hi,
I will explain the problem I am facing first, then ask a couple of questions on how to tweak logstash.

1)The Problem:
Logstash filters are not working after a while. Logstash only understands the @type, but none of the filters are working.

Output of kibana showing both working and a broken one is in the attachment.

I believe logstash restarts and it does not work, because previously I was using an experimental plugin and experimental warning was in the outputs.

I blamed the experimental plugin for this bug, but it still happens after removing it.

----------------------------------------------

2) When I check the memory usage I see that logstash is using 600 MB.

a) Could this be the issue? Would starting logstash with Java's -Xms and -Xmx options work ?

b) I am working on 2 different types of documents. Logcat and apache logs. Would it be better if I split them in different config files and start 2 different logstash instances?

c) Logstash working with a single accumulating log file is a well tested usage. (like apache access logs)

With android logcat logs, they come as a new document, and does not append to the previous one, hence there would be too many documents. Should I use a diffrent strategy, like deleting old files after a while, so that logstash does not have to listen changes on those documents?

Thanks in advance.
Ali

Attachments

2

Gliffy Diagrams

Activity

Show:

Philippe Weber August 1, 2014 at 12:00 PM

A little more aggressive garbage-collect on jira.com, expecting follow-up activities to take place on github issues

Jason Kendall March 5, 2014 at 4:11 PM

Is the filtering issue still happening in 1.3.3?

The questions are best served in the mailing list, or IRC.

REJECTED

Details

Assignee

Reporter

Affects versions

Created April 19, 2013 at 4:24 PM
Updated August 1, 2014 at 12:00 PM
Resolved August 1, 2014 at 12:00 PM