elasticsearch throws exceptions when logstash is running in standalone mode
Description
Gliffy Diagrams
Activity
Jordan Sissel May 5, 2013 at 1:42 AM
Sunny Jaisinghani April 3, 2013 at 12:15 PM
I gave up on the embedded version of elasticsearch ( 0.20.2 ) and used elasticsearch ver 0.20.6 separately with default configs.
Using elasticsearch this way really helps. All my apps, logstash, elasticsearch and kibana are working in harmony
None of them crashes.
For more details, refer
http://edgeofsanity.net/article/2012/12/26/elasticsearch-for-logging.html
https://github.com/tavisto/elasticsearch-rpms
Utkarsh Sengar April 2, 2013 at 6:53 PM
I suddenly started facing the same error after I introduced a new node in ES cluster. The existing ES logstash node automatically discovered the new node via zen discovery. But then, I pulled down the new node.
After that, I have tried restarting the logstash elasticsearch node but it not recovering from this error.
Sunny Jaisinghani March 25, 2013 at 11:54 AM
Logstash developers,
Would somebody please help me out with a solution to the error reported in the initial post.
Thanks
Sunny
Sunny Jaisinghani March 4, 2013 at 11:04 AM
Here is one more ES exception seen in logstash logs
{:message=>"Failed to index an event, will retry", :exception=>org.elasticsearch.action.UnavailableShardsException: [logstash-2013.03.04][2] [2] shardIt, [1] active : Timeout waiting for [1m],
Hello,
I have recently started exploring logstash after i heard it from my peers who attended puppetconf 2012
We have deployed logstash 1.1.9 in standalone mode with embedded elasticsearch on a EC2 instance.
We have also installed kibana as a frontend for logstash.
Since, we are using the embedded version of elasticsearch, i assume it doesn't have any compatibility issues with logstash version.
{:message=>"Failed to index an event, will retry", :exception=>org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [1m], :event=>{"@source"=>"file://ip-10-xx-xx-xx.ec2.internal/var/log/messages", "@tags"=>[], "@fields"=>{}, "@timestamp"=>"2013-03-01T09:08:39.998Z", "@source_host"=>"ip-10-157-xx-xx.ec2.internal", "@source_path"=>"/var/log/messages", "@message"=>"Mar 1 00:14:52 ip-10-122-70-221@type"=>"linuxsyslog"}, :level=>:warn}
I am also seeing following exception
{:message=>"Failed to index an event, will retry", :exception=>org.elasticsearch.transport.RemoteTransportException: [Imperial Hydra][inet/10.157.38.34:9300][index], :event=>{"@source"=>"file://ip-10-xx-xx-xx.ec2.internal/var/log/", "@tags"=>[], "@fields"=>{}, "@timestamp"=>"2013-02-27T13:49:13.515Z", "@source_host"=>"ip-10-xx-xx-xx.ec2.internal", "@source_path"=>"/var/log/",
{:message=>"Failed to index an event, will retry", :exception=>org.elasticsearch.transport.RemoteTransportException: [Ariann]
{:message=>"Failed to index an event, will retry", :exception=>org.elasticsearch.transport.RemoteTransportException: [Gorgon]
{:message=>"Failed to index an event, will retry", :exception=>org.elasticsearch.transport.RemoteTransportException: [Forgotten One]
following processes are running on the host
root 22155 1 1 00:11 pts/0 00:01:52 /usr/bin/java -jar /opt/logstash/bin/logstash.jar agent --config /opt/logstash/conf/logstash.conf --log /opt/logstash/log/logstash.log --grok-patterns-path /opt/logstash/patterns
root 22039 1 0 00:08 ? 00:00:01 ruby kibana.rb
root 28932 13009 0 02:05 pts/0 00:00:00 grep -i java
tcp 0 0 0.0.0.0:9200 0.0.0.0:* LISTEN 22155/java
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 22039/ruby
tcp 0 0 0.0.0.0:9300 0.0.0.0:* LISTEN 22155/java
tcp 0 0 0.0.0.0:9301 0.0.0.0:* LISTEN 22155/java
logstash.conf
================
input {
file {
type => "linuxsyslog"
Wildcards work, here
path => [ "/var/log/messages*" ]
sincedb_path => "/opt/logstash"
}
file {
type => "merrors"
path => [ "/var/log/m.errors*" ]
sincedb_path => "/opt/logstash"
}
file {
type => "mall"
path => [ "/var/log/m*" ]
exclude => [ "/var/log/m.errors*" ]
sincedb_path => "/opt/logstash"
}
}
filter {
date {
type => "syslog"
syslog_timestamp => [ "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
output {
elasticsearch {
embedded => true
host => "10.157.3x.3x"
}
}
If i restart logstash and try to use the search engine, it works and gives output for a while, before it starts complaining about the above errors.
Please suggest on how to fix these errors.
Thanks