How to troubleshoot Exception: java.lang.ThreadDeath

Description

Since putting a custom plugin in place, that make an external call to a REST service, I have been getting redis queue alerts.

The redis queue typically maintains a 0 queue, but recently it spikes to 14,000 then back down to 0 in about 2 minutes.

I have a feeling it has to do with my new plugin. Is there a way to get more details about the following error without running with -vvvv

Exception: java.lang.ThreadDeath thrown from the UncaughtExceptionHandler in thread "Thread-45"

What thread is throwing the exception.

Activity

Show:
Zachary Buckholz
July 10, 2014, 3:06 PM

I was able to capture some additional details this morning.

As I expected it was pointing to my new custom plugin. I should probably wrap the rest call in a catch try block.

I will make this change and update this for tracking my issue.

Exception: java.lang.ThreadDeath thrown from the UncaughtExceptionHandler in thread "Thread-45"

Exception in filterworker {"exception"=>Request Timeout: , "backtrace"=>["file:/home/lsapp/logstash-1.3.2-flatjar.jar!/restclient/request.rb:184:in `transmit'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/restclient/request.rb:172:in `t
ransmit'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/restclient/request.rb:64:in `execute'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/restclient/request.rb:33:in `execute'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/restclien
t/resource.rb:54:in `get'", "/home/lsapp/logstash/build/ruby/logstash/filters/crowd.rb:67:in `filter'", "(eval):167:in `initialize'", "org/jruby/RubyProc.java:271:in `call'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/logstash/pipeline
.rb:255:in `filter'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/logstash/pipeline.rb:196:in `filterworker'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/logstash/pipeline.rb:136:in `start_filters'"], :level=>:error}

Exception: java.lang.ThreadDeath thrown from the UncaughtExceptionHandler in thread "Thread-243"

Exception: java.lang.ThreadDeath thrown from the UncaughtExceptionHandler in thread "Thread-282"

Exception: java.lang.ThreadDeath thrown from the UncaughtExceptionHandler in thread "Thread-291"
Exception in filterworker {"exception"=>Request Timeout: , "backtrace"=>["file:/home/lsapp/logstash-1.3.2-flatjar.jar!/restclient/request.rb:184:in `transmit'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/restclient/request.rb:172:in `t
ransmit'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/restclient/request.rb:64:in `execute'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/restclient/request.rb:33:in `execute'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/restclien
t/resource.rb:54:in `get'", "/home/lsapp/logstash/build/ruby/logstash/filters/crowd.rb:67:in `filter'", "(eval):167:in `initialize'", "org/jruby/RubyProc.java:271:in `call'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/logstash/pipeline
.rb:255:in `filter'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/logstash/pipeline.rb:196:in `filterworker'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/logstash/pipeline.rb:136:in `start_filters'"], :level=>:error}
Exception in filterworker {"exception"=>Request Timeout: , "backtrace"=>["file:/home/lsapp/logstash-1.3.2-flatjar.jar!/restclient/request.rb:184:in `transmit'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/restclient/request.rb:172:in `t
ransmit'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/restclient/request.rb:64:in `execute'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/restclient/request.rb:33:in `execute'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/restclien
t/resource.rb:54:in `get'", "/home/lsapp/logstash/build/ruby/logstash/filters/crowd.rb:67:in `filter'", "(eval):167:in `initialize'", "org/jruby/RubyProc.java:271:in `call'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/logstash/pipeline
.rb:255:in `filter'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/logstash/pipeline.rb:196:in `filterworker'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/logstash/pipeline.rb:136:in `start_filters'"], :level=>:error}

Exception: java.lang.ThreadDeath thrown from the UncaughtExceptionHandler in thread "Thread-330"
Exception in filterworker {"exception"=>Request Timeout: , "backtrace"=>["file:/home/lsapp/logstash-1.3.2-flatjar.jar!/restclient/request.rb:184:in `transmit'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/restclient/request.rb:172:in `t
ransmit'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/restclient/request.rb:64:in `execute'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/restclient/request.rb:33:in `execute'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/restclien
t/resource.rb:54:in `get'", "/home/lsapp/logstash/build/ruby/logstash/filters/crowd.rb:67:in `filter'", "(eval):167:in `initialize'", "org/jruby/RubyProc.java:271:in `call'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/logstash/pipeline
.rb:255:in `filter'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/logstash/pipeline.rb:196:in `filterworker'", "file:/home/lsapp/logstash-1.3.2-flatjar.jar!/logstash/pipeline.rb:136:in `start_filters'"], :level=>:error}

Zachary Buckholz
July 11, 2014, 12:55 PM

I am not sure what's causing this. I was able to attach jconsole to logstash and monitor for several hours, but nothing stood out.

Is there anything else I can do in my ruby plugin code to isolate this? It's obviously coming from my plugin since that's the only new thing, and a minor change to a grok pattern.

Thanks
Zach

Zachary Buckholz
July 12, 2014, 3:21 AM

I wrapped several blocks of code in try / catch blocks. So far no crashing issues.

Assignee

Logstash Developers

Reporter

Zachary Buckholz

Labels

None

Affects versions

Configure