ElasticSearch and Logstash Integration Error
Description
Gliffy Diagrams
Activity
Show:

Jonathan Moo December 7, 2011 at 3:28 AM
Yes the elasticsearch.jar finally compiled. Thanks!
Jordan Sissel December 6, 2011 at 10:25 AM
Sounds like logstash can't find your elasticsearch jar.
Best recommendation is to use the logstash release jar files - they contain pretty much all the dependencies and are easy to run with simply 'java -jar logstash.jar ...'
Is this OK?
According to the installation guide, I have installed the version 0.17.6 of ElasticSearch and after trying to run Logstash, the following error occurred:
~/Applications/logstash_conf$ logstash -f agent-server.conf
NameError: cannot load Java class org.elasticsearch.action.ActionListener
get_proxy_or_package_under_package at org/jruby/javasupport/JavaUtilities.java:54
method_missing at /home/jon1984/.rvm/rubies/jruby-1.6.5/lib/ruby/site_ruby/shared/builtin/javasupport/java.rb:51
ActionListener at /home/jon1984/.rvm/gems/jruby-1.6.5/gems/jruby-elasticsearch-0.0.12/lib/jruby-elasticsearch/actionlistener.rb:5
(root) at /home/jon1984/.rvm/gems/jruby-1.6.5/gems/jruby-elasticsearch-0.0.12/lib/jruby-elasticsearch/actionlistener.rb:4
require at org/jruby/RubyKernel.java:1038
require at /home/jon1984/.rvm/rubies/jruby-1.6.5/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36
(root) at /home/jon1984/.rvm/gems/jruby-1.6.5/gems/jruby-elasticsearch-0.0.12/lib/jruby-elasticsearch/actionlistener.rb:2
require at org/jruby/RubyKernel.java:1038
require at /home/jon1984/.rvm/rubies/jruby-1.6.5/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36
(root) at /home/jon1984/.rvm/gems/jruby-1.6.5/gems/jruby-elasticsearch-0.0.12/lib/jruby-elasticsearch/request.rb:2
require at org/jruby/RubyKernel.java:1038
require at /home/jon1984/.rvm/rubies/jruby-1.6.5/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36
(root) at /home/jon1984/.rvm/gems/jruby-1.6.5/gems/jruby-elasticsearch-0.0.12/lib/jruby-elasticsearch/indexrequest.rb:3
require at org/jruby/RubyKernel.java:1038
require at /home/jon1984/.rvm/rubies/jruby-1.6.5/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36
(root) at /home/jon1984/.rvm/gems/jruby-1.6.5/gems/jruby-elasticsearch-0.0.12/lib/jruby-elasticsearch/client.rb:1
require at org/jruby/RubyKernel.java:1038
require at /home/jon1984/.rvm/rubies/jruby-1.6.5/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36
register at /home/jon1984/.rvm/gems/jruby-1.6.5/gems/jruby-elasticsearch-0.0.12/lib/jruby-elasticsearch.rb:81
run_output at /home/jon1984/.rvm/gems/jruby-1.6.5/gems/logstash-1.0.17/lib/logstash/agent.rb:522
run_with_config at /home/jon1984/.rvm/gems/jruby-1.6.5/gems/logstash-1.0.17/lib/logstash/agent.rb:370
call at org/jruby/RubyProc.java:270
call at org/jruby/RubyProc.java:224
This is the current agent-server.conf file:
input {
amqp {
ship logs to the 'rawlogs' fanout queue.
type => "all"
host => "172.17.112.98"
exchange_type => "fanout"
name => "rawlogs"
}
}
filter {
grok {
type => "syslog" # for logs of type "syslog"
pattern => "%{SYSLOGLINE}"
You can specify multiple 'pattern' lines
}
grok {
type => "apache-access" # for logs of type 'apache-access'
pattern => "%{COMBINEDAPACHELOG}"
}
date {
type => "syslog"
The 'timestamp' and 'timestamp8601' names are for fields in the
logstash event. The 'SYSLOGLINE' grok pattern above includes a field
named 'timestamp' that is set to the normal syslog timestamp if it
exists in the event.
timestamp => "MMM d HH:mm:ss" # syslog 'day' value can be space-leading
timestamp => "MMM dd HH:mm:ss"
timestamp8601 => ISO8601 # Some syslogs use ISO8601 time format
}
date {
type => "apache-access"
timestamp => "dd/MMM/yyyy:HH:mm:ss Z"
}
}
output {
stdout { }
If your elasticsearch server is discoverable with multicast, use this:
elasticsearch { }
If you can't discover using multicast, set the address explicitly
elasticsearch {
host => "172.17.112.98"
port => "9301"
}
}
I have also installed the river rabbitmq plugin for Elastic Search.
Logstash centralised setting works well without Elastic Search, as I can see the log output from the other servers. However I want to integrate Elastic Search with it, so any idea what has gone wrong?
Many thanks!
Regards,
Jonathan