Logfile growth
Description
Attachments
- 19 Apr 2013, 04:07 PM
Gliffy Diagrams
Activity
Philippe Weber September 30, 2013 at 12:12 PM
Garbage collect old support ticket
Eugene Taranov April 19, 2013 at 6:44 PM
Yes, it was up and running all the time, however it might not be responding over http while doing GC, will try to monitor this.
Sorry, the only exception I have is at the beginning of the log I posted earlier {:message=>"Output thread exception", 
lugin=>#<LogStash::Outputs::ElasticSearchHTTP:0x5082f7ff ..., starting from ~550000 char it contains a dump of application logs until the end of file. Possibly, it got truncated since partition run out of space.
Jordan Sissel April 19, 2013 at 4:41 PM
Hmm, seems like logstash is having trouble talking to elasticsearch. Is it up? The error message there is missing some information, there's usually an exception listed on the line above what you posted if you can find that?
Eugene Taranov April 19, 2013 at 4:08 PM
Jordan,
attached head of log file truncating rest of web application related stuff.
Thank you.
Jordan Sissel April 19, 2013 at 3:51 PM
It would be helpful to know what is in the log, otherwise I don't know what to look into 
Since switch to external elasticsearch logstash's log file constantly grows by several GB every day; debug output is not enabled; disabling of log file or directing to it to the /dev/null did not work.
Config:
input {
udp {
port => 1514
type => "nginx-access"
}
udp {
port => 1515
type => "nginx-error"
}
udp {
port => 1516
type => "mail"
}
}
filter {
rotocol}\" %{POSINT:response_code} %{INT:response_size} %{QS:referrer} %{QS:user_agent}"
mutate {
remove => [ "@source_path", "@source" ]
rename => [ "@source_host", "syslog" ]
}
grok {
type => "nginx-access"
pattern => "%{SYSLOGTIMESTAMP} %{HOSTNAME:webnode} \"\" %{HOSTNAME} %{IPORHOST:vhost} %{DATA} %{DATA} %{DATA} %{IP:client} [%{HTTPDATE}] %{DATA} %{DATA} \"%{DATA:request_type} %{DATA:request} %{DATA
}
grok {
type => "nginx-error"
pattern => "%{SYSLOGTIMESTAMP} %{HOSTNAME:webnode} %{GREEDYDATA:error}"
}
}
output {
elasticsearch_http { }
}
/usr/bin
/java -Xmx512M -Xms512M -jar logstash-1.1.9-monolithic.jar agent --config logstash.conf --log log/logstash.log
Please look into this.