logstash crash when elasticsearch output no longer reachable

Description

First off this is for v1.4.2 but that's not listed in the version dropdown.

I have two ES nodes that are 2 different ES clusters that logstash sends indexes to - one for all indexes and one just for web indexes:

output {
elasticsearch {
cluster => "all"
node_name => "logstash"
host => "all-indexes"
port => 9300
protocol => 'node'
}
if ("apache" in [tags]) {
elasticsearch {
cluster => "web"
node_name => "logstash"
host => "web-indexes"
port => 9300
protocol => 'node'
}
}
}

Both are ES 1.1.1. Yesterday I had a kernel update to push out on the web-indexes machine, so I rebooted. When I did, logstash eventually crashed because it could no longer talk to it:

log4j, [2014-08-05T13:53:34.699] WARN: org.elasticsearch.discovery.zen: [web-indexes] master_left and no other node elected to become master, current nodes: {[web-indexes][IWa9aAqyTAiR2jhXHMesDA][all-indexes][inet/192.168.0.2:9302]{client=true, data=false},}
log4j, [2014-08-05T15:18:47.745] WARN: org.elasticsearch.discovery.zen: [web-indexes] master_left and no other node elected to become master, current nodes: {[web-indexes][IWa9aAqyTAiR2jhXHMesDA][all-indexes][inet/192.168.0.2:9302]{client=true, data=false},}
log4j, [2014-08-05T15:19:38.860] WARN: org.elasticsearch.transport.netty: [web-indexes] exception caught on transport layer [[id: 0xb3379c8b]], closing connection
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

That traceback is repeated several more times before logstash finally crashed for good. Restarting logstash when both ES servers were online again worked

Environment

None

Status

Assignee

Logstash Developers

Reporter

Ti Leggett

Labels

Affects versions

1.4.0

Priority