logstash crash when elasticsearch output no longer reachable

Description

First off this is for v1.4.2 but that's not listed in the version dropdown.

I have two ES nodes that are 2 different ES clusters that logstash sends indexes to - one for all indexes and one just for web indexes:

output {
elasticsearch {
cluster => "all"
node_name => "logstash"
host => "all-indexes"
port => 9300
protocol => 'node'
}
if ("apache" in [tags]) {
elasticsearch {
cluster => "web"
node_name => "logstash"
host => "web-indexes"
port => 9300
protocol => 'node'
}
}
}

Both are ES 1.1.1. Yesterday I had a kernel update to push out on the web-indexes machine, so I rebooted. When I did, logstash eventually crashed because it could no longer talk to it:

log4j, [2014-08-05T13:53:34.699] WARN: org.elasticsearch.discovery.zen: [web-indexes] master_left and no other node elected to become master, current nodes: {[web-indexes][IWa9aAqyTAiR2jhXHMesDA][all-indexes][inet/192.168.0.2:9302]{client=true, data=false},}
log4j, [2014-08-05T15:18:47.745] WARN: org.elasticsearch.discovery.zen: [web-indexes] master_left and no other node elected to become master, current nodes: {[web-indexes][IWa9aAqyTAiR2jhXHMesDA][all-indexes][inet/192.168.0.2:9302]{client=true, data=false},}
log4j, [2014-08-05T15:19:38.860] WARN: org.elasticsearch.transport.netty: [web-indexes] exception caught on transport layer [[id: 0xb3379c8b]], closing connection
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

That traceback is repeated several more times before logstash finally crashed for good. Restarting logstash when both ES servers were online again worked

Activity

Show:
Kurt Hurtado
February 6, 2015, 10:07 PM

Assignee

Logstash Developers

Reporter

Ti Leggett

Labels

Affects versions

Configure