In an attempt to address issues with AMQP performance (mainly around consumer side), I've done tentative work on moving JRuby logstash to use the hot_bunnies gem. This has been made configurable as well with a new optional configuration declaration:
Again, if you are on JRuby, the default is to use hot_bunnies. This is a JRuby gem that uses the RabbitMQ client jar.
You can track my fork for additional changes to run from source.
Additionally, I've added a monolithic jar for folks to make testing easier. You can download it here
If you have felt like AMQP performance (especially on the input side) have been subpar, please test this jar and help me work out the kinks.
Some things to note/limitations:
SSL is not exposed in hot_bunnies yet. I plan on submitting pull requests for that.
By default, logstash sets message acknowledgements on output to true. This has a considerable performance impact regardless of using bunny or hot_bunnies.
To help with debugging, if you can run with the '-v' option, that would be helpful. Please paste the output from the metrics dump in this ticket.
If possible, install the rabbitmq web management plugin and restart your broker. This will help you get real-time stats from the broker as well.
In most cases it's fairly sane to disable message acknowledgements (ack => false) however you need to understand the imapact. By disabling acks, there is a small chance that, in the event that something goes wrong on the logstash side you will get double messages. Logstash plays it safe by explicitly setting this to true while the drivers themselves default to false.
If you run with '-v' you will get periodic metric dumps from the cabin logger. The divider between dumps is the 'message dump' stanza itself. A full dump for amqp output looks like this in this build:
A dump for the amqp input looks something like this:
You can read how to install it here:
It will provide a nice gui based interface for seeing realtime message rates inside the broker.
With the new codec stuff coming and the ruby-amqp gem being faster these days, I don't think hot bunnies will be necessary.