when using syslog input, logstash will crash on high concurruncy

Description

when using syslog input on high concurruncy as 200+, uncatched error Errno::EBADF(bad file descriptor) will crash logstash.

maybe Errno::EBADF will also be rescued in syslog.rb tcp_listener like:

syslog.rb
143 begin
144 client.each do |line|
145 @codec.decode(line) do |event|
146 decorate(event)
147 event["host"] = ip
148 syslog_relay(event)
149 output_queue << event
150 end
151 end
152 rescue Errno::EBADF
153 rescue Errno::ECONNRESET
154 end

Activity

Show:
October 11, 2013, 3:07 AM

error message:

it will meet IOError sometimes

Bruno Galindro da Costa
February 25, 2014, 12:11 PM

I suggest you to run logstash in debug mode and see when the problem occurs:

With me, the same thing occurred and I've noticed that Logstash entered in exception when it tried to execute this command:

So, the problem is that my ElasticSearch cluster was down. After started, the problem was gone.

BUT, I suggest to the development team to put a friendly message in logstash when things like these occurs.

Tim Bunce
February 27, 2014, 12:47 PM

Just a note to apologise for editing the labels on the case. I found this case via google as the original poster had the same symptom (Errno::EBADF crashing logstash) and Bruno's reply matched my experience (cause was elasticsearch being down). I started writing up a comment but then realised the original posters issue was different (syslog). So I've opened a new case LOGSTASH-1948.

Assignee

Logstash Developers

Reporter

Configure