Replace bunny with hot_bunnies by default when running under JRuby

Description

In an attempt to address issues with AMQP performance (mainly around consumer side), I've done tentative work on moving JRuby logstash to use the hot_bunnies gem. This has been made configurable as well with a new optional configuration declaration:

or

Again, if you are on JRuby, the default is to use hot_bunnies. This is a JRuby gem that uses the RabbitMQ client jar.

You can track my fork for additional changes to run from source.

Additionally, I've added a monolithic jar for folks to make testing easier. You can download it here

If you have felt like AMQP performance (especially on the input side) have been subpar, please test this jar and help me work out the kinks.

Some things to note/limitations:

  • SSL is not exposed in hot_bunnies yet. I plan on submitting pull requests for that.

  • By default, logstash sets message acknowledgements on output to true. This has a considerable performance impact regardless of using bunny or hot_bunnies.

  • To help with debugging, if you can run with the '-v' option, that would be helpful. Please paste the output from the metrics dump in this ticket.

  • If possible, install the rabbitmq web management plugin and restart your broker. This will help you get real-time stats from the broker as well.

Message Acknowledgements

In most cases it's fairly sane to disable message acknowledgements (ack => false) however you need to understand the imapact. By disabling acks, there is a small chance that, in the event that something goes wrong on the logstash side you will get double messages. Logstash plays it safe by explicitly setting this to true while the drivers themselves default to false.

Metric dumps

If you run with '-v' you will get periodic metric dumps from the cabin logger. The divider between dumps is the 'message dump' stanza itself. A full dump for amqp output looks like this in this build:

A dump for the amqp input looks something like this:

Management plugin

You can read how to install it here:

http://www.rabbitmq.com/management.html

It will provide a nice gui based interface for seeing realtime message rates inside the broker.

Assignee

John E. Vincent

Reporter

John E. Vincent

Labels

None

Affects versions

Configure