Many accumulating open connections

Description

My logstash process has crashed with the error 'too many open files'. Once restarted, it has slowly been accumulating more open files, which are actually tcp sockets that seem to not be closing. The client sending messages to logstash is rsyslog, on port 514 (normal syslog messages.) I have quite a few of these rsyslog clients, but only one of them seems to have these connections remaining open.

My logstash config file:
https://gist.github.com/ea5587d94e2cfdfed527

Stack trace from last crash:
https://gist.github.com/eab424c23617aa31807a

lsof -p <logstash_pid> output:
https://gist.github.com/f884f94b4da2e821d38a

Activity

Show:
Jordan Sissel
March 13, 2012, 12:49 AM

I'm pretty sure this is because closed client sockets are left dangling and un-garbagecollected so they are never closed.

Two things:

  • evaluate whether or not we care about tracking the list of all client sockets (@tcp_clients in inputs/syslog.rb)

  • close the socket on any read error.

Richard Pijnenburg
December 17, 2012, 5:06 AM

Syslog input plugin will be removed.
See: https://logstash.jira.com/browse/LOGSTASH-427

Andrei Chiriaev
October 30, 2014, 9:37 AM

Guys, Hi. Had the same issue with 1.4.2. Anyone else reporting similar issues? Regards, Andrei
...
{:timestamp=>"2014-10-29T21:58:06.532000+0000", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::Tcp type=>\"nxlog_json\", tags=>[\"input_nxlog_json\"], host=>\"0.0.0.0\", mode=>\"server\">\n Error: Too many open files", :level=>:error}
{:timestamp=>"2014-10-29T21:58:10.704000+0000", :message=>"An error occurred. Closing connection", :client=>"10.51.4.7:55655", :exception=>#<IOError: Too many open files>, :backtrace=>["org/jruby/RubyIO.java:2992:in `sysread'", "/opt/logstash/lib/logstash/inputs/tcp.rb:162:in `read'", "/opt/logstash/lib/logstash/inputs/tcp.rb:111:in `handle_socket'", "/opt/logstash/lib/logstash/inputs/tcp.rb:146:in `client_thread'", "/opt/logstash/lib/logstash/inputs/tcp.rb:144:in `client_thread'"], :level=>:error}
...

Aaron Mildenstein
February 6, 2015, 7:37 PM
Fixed

Assignee

Logstash Developers

Reporter

Tim

Labels

Fix versions

Affects versions

Configure