Logfile growth

Description

Since switch to external elasticsearch logstash's log file constantly grows by several GB every day; debug output is not enabled; disabling of log file or directing to it to the /dev/null did not work.

Config:
input {
udp {
port => 1514
type => "nginx-access"
}
udp {
port => 1515
type => "nginx-error"
}
udp {
port => 1516
type => "mail"
}
}

filter {
mutate {
remove => [ "@source_path", "@source" ]
rename => [ "@source_host", "syslog" ]
}
grok {
type => "nginx-access"
pattern => "%{SYSLOGTIMESTAMP} %{HOSTNAME:webnode} \"\" %{HOSTNAME} %{IPORHOST:vhost} %{DATA} %{DATA} %{DATA} %{IP:client} [%{HTTPDATE}] %{DATA} %{DATA} \"%{DATA:request_type} %{DATA:request} %{DATAface with tonguerotocol}\" %{POSINT:response_code} %{INT:response_size} %{QS:referrer} %{QS:user_agent}"
}
grok {
type => "nginx-error"
pattern => "%{SYSLOGTIMESTAMP} %{HOSTNAME:webnode} %{GREEDYDATA:error}"
}
}

output {
elasticsearch_http { }
}

/usr/bin
/java -Xmx512M -Xms512M -jar logstash-1.1.9-monolithic.jar agent --config logstash.conf --log log/logstash.log

Please look into this.

Attachments

1
  • 19 Apr 2013, 04:07 PM

Gliffy Diagrams

Activity

Show:

Philippe Weber September 30, 2013 at 12:12 PM

Garbage collect old support ticket

Eugene Taranov April 19, 2013 at 6:44 PM

Yes, it was up and running all the time, however it might not be responding over http while doing GC, will try to monitor this.
Sorry, the only exception I have is at the beginning of the log I posted earlier {:message=>"Output thread exception", face with tonguelugin=>#<LogStash::Outputs::ElasticSearchHTTP:0x5082f7ff ..., starting from ~550000 char it contains a dump of application logs until the end of file. Possibly, it got truncated since partition run out of space.

Jordan Sissel April 19, 2013 at 4:41 PM

Hmm, seems like logstash is having trouble talking to elasticsearch. Is it up? The error message there is missing some information, there's usually an exception listed on the line above what you posted if you can find that?

Eugene Taranov April 19, 2013 at 4:08 PM

Jordan,

attached head of log file truncating rest of web application related stuff.

Thank you.

Jordan Sissel April 19, 2013 at 3:51 PM

It would be helpful to know what is in the log, otherwise I don't know what to look into slightly smiling face

Incomplete

Details

Assignee

Reporter

Affects versions

Created April 19, 2013 at 2:55 PM
Updated September 30, 2013 at 12:12 PM
Resolved September 30, 2013 at 12:12 PM