Gelfd TooManyChunksError - gelf listener died

Description

Environment:
logstash-1.1.13-flatjar.jar
elasticsearch 0.20.6
redis-2.6.11
FreeBSD 9.1-RELEASE-p3

Windows 2k8 server with .net gelf
GELF 1.0.2.12

input {
gelf {
debug => true

  1. host => "0.0.0.0"
    format => plain
    port => 5515
    type => gelf
    }
    }
    output {
    redis {
    host => "127.0.0.1"

  2. type => "redis-input"

  3. these settings should match the output of the agent
    data_type => "list"
    key => "logstash"

  4. We use json_event here since the sender is a logstash agent

  5. message_format => "json_event"
    }
    }

logstash log:

{:timestamp=>"2013-08-06T11:42:41.650000-0700", :message=>"gelf listener died", :exception=>#<Gelfd::TooManyChunksError: 166 greater than 128>, :backtrace=>["file:/usr/local/logstash/logstash-1.1.13-flatjar.jar!/gelfd/chunked_parser.rb:40:in `parse_chunk'", "file:/usr/local/logstash/logstash-1.1.13-flatjar.jar!/gelfd/chunked_parser.rb:8:in `parse'", "file:/usr/local/logstash/logstash-1.1.13-flatjar.jar!/gelfd/parser.rb:10:in `parse'", "file:/usr/local/logstash/logstash-1.1.13-flatjar.jar!/logstash/inputs/gelf.rb:83:in `udp_listener'", "org/jruby/RubyKernel.java:1409:in `loop'", "file:/usr/local/logstash/logstash-1.1.13-flatjar.jar!/logstash/inputs/gelf.rb:79:in `udp_listener'", "file:/usr/local/logstash/logstash-1.1.13-flatjar.jar!/logstash/inputs/gelf.rb:59:in `run'", "file:/usr/local/logstash/logstash-1.1.13-flatjar.jar!/logstash/agent.rb:761:in `run_input'", "file:/usr/local/logstash/logstash-1.1.13-flatjar.jar!/logstash/agent.rb:407:in `start_input'"], :level=>:warn}

Gliffy Diagrams

Activity

Show:

Dan Ballinger August 27, 2013 at 10:49 PM
Edited

In Summary:

1) Too many chunks error causes the gelfd listener to die.
2) The listener tries to restart using the udp_listener.
3) There appears to be an issue with the udp_listener restart in inputs/gelf.rb

Dan Ballinger August 27, 2013 at 10:41 PM
Edited

This message is followed by the following in 5 second intervals:

{:timestamp=>"2013-08-06T11:42:46.666000-0700", :message=>"gelf listener died", :exception=>#<IOError: not opened for reading>, :backtrace=>["org/jruby/ext/socket/RubyBasicSocket.java:410:in `close_read'", "file:/usr/local/logstash/logstash-1.1.13-flatjar.jar!/logstash/inputs/gelf.rb:72:in `udp_listener'", "file:/usr/local/logstash/logstash-1.1.13-flatjar.jar!/logstash/inputs/gelf.rb:59:in `run'", "file:/usr/local/logstash/logstash-1.1.13-flatjar.jar!/logstash/agent.rb:761:in `run_input'", "file:/usr/local/logstash/logstash-1.1.13-flatjar.jar!/logstash/agent.rb:407:in `start_input'"], :level=>:warn}

Details

Assignee

Reporter

Affects versions

Created August 6, 2013 at 8:22 PM
Updated February 14, 2015 at 5:55 AM