lumberjack input threads stuck processing ssl

Description

  1. ps auxw | grep -i logstash
    root 29519 29.0 36.0 13359960 2525604 pts/2 Sl 01:55 8:01 /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java -Xmx8g -Xms2g -jar /opt/logstash/logstash-1.3.3-flatjar.jar agent --filterworkers 4 -f /etc/logstash/indexer.conf -l /var/log/logstash/indexer.log

A healthy logstash instance spends little time on ssl:

  1. jstack -l 29519 | grep org.jruby.ext.openssl.SSLSocket.doHandshake
    XXX nothing

and has about 100 sockets in CLOSE_WAIT:

netstat -pant | grep CLOSE_WAIT | grep 5005 | wc -l
112

over time, the lumberjack input threads spend more and more time on ssl and the number of CLOSE_WAIT sockets goes up 4x to 6x

  1. grep org.jruby.ext.openssl.SSLSocket.doHandshake jstack.log | wc -l
    12

  1. grep org.jruby.ext.openssl.SSLSocket.doHandshake jstack2.log | wc -l
    13

  1. grep org.jruby.ext.openssl.SSLSocket.doHandshake jstack3.log | wc -l
    14

I found this jruby issue, which sounds similar:
https://github.com/jruby/jruby/issues/1156

on an 8 core server, the cpu utilization goes up in very distinct 1/8 block steps indicating that one core is being used and then very suddenly a second core is used fully, then a third etc...

If there is no fix, then I would like to request the ability to use stunnel for ssl and pass plain text to the lumberjack input plugin for logstash.

Activity

Show:
Lee Johnson
February 4, 2014, 3:34 PM

Probably related to

Assignee

Logstash Developers

Reporter

Donald Endres

Labels

Affects versions

Configure