Uploaded image for project: 'logstash'
  1. LOGSTASH-2275

logstash crash when elasticsearch output no longer reachable

    Details

      Description

      First off this is for v1.4.2 but that's not listed in the version dropdown.

      I have two ES nodes that are 2 different ES clusters that logstash sends indexes to - one for all indexes and one just for web indexes:

      output {
      elasticsearch

      { cluster => "all" node_name => "logstash" host => "all-indexes" port => 9300 protocol => 'node' }

      if ("apache" in [tags]) {
      elasticsearch

      { cluster => "web" node_name => "logstash" host => "web-indexes" port => 9300 protocol => 'node' }

      }
      }

      Both are ES 1.1.1. Yesterday I had a kernel update to push out on the web-indexes machine, so I rebooted. When I did, logstash eventually crashed because it could no longer talk to it:

      log4j, [2014-08-05T13:53:34.699] WARN: org.elasticsearch.discovery.zen: [web-indexes] master_left and no other node elected to become master, current nodes: {[web-indexes][IWa9aAqyTAiR2jhXHMesDA][all-indexes][inet[/192.168.0.2:9302]]

      {client=true, data=false},}
      log4j, [2014-08-05T15:18:47.745] WARN: org.elasticsearch.discovery.zen: [web-indexes] master_left and no other node elected to become master, current nodes: {[web-indexes][IWa9aAqyTAiR2jhXHMesDA][all-indexes][inet[/192.168.0.2:9302]]{client=true, data=false}

      ,}
      log4j, [2014-08-05T15:19:38.860] WARN: org.elasticsearch.transport.netty: [web-indexes] exception caught on transport layer [[id: 0xb3379c8b]], closing connection
      java.net.NoRouteToHostException: No route to host
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
      at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150)
      at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
      at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
      at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
      at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
      at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
      at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)

      That traceback is repeated several more times before logstash finally crashed for good. Restarting logstash when both ES servers were online again worked

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                Logstash Developers (Inactive)
                Reporter:
                Ti Leggett
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: