RabbitMQ 3.2.2 and Logstash 1.3.2 key => "%{origin}"

Description

I have an odd situation relating to an upgrade from 1.2.2 to 1.3.2. Specifically, Logstash 1.3.2 will not connect to RabbitMQ 3.2.2 when the key looks like this key => "%{origin}". It works perfectly in 1.2.2

However, changing the key to something like key => "FOO" will work without issue in 1.3.2. Is this a bug or broader change in 1.3.2 that I missed?

Here is my Rabbit output block which works in 1.2.2

rabbitmq {
debug => false
codec => "json"
durable => true
exchange_type => "direct"
host => "HOST"
key => "%{origin}"
exchange => "api.postworker"
persistent => true
port => 5670
user => "USER"
password => "PASS"
#workers => 4
}

This works in 1.3.2

rabbitmq {
debug => false
codec => "json"
durable => true
exchange_type => "direct"
host => "HOST"
key => "FOO"
exchange => "api.postworker"
persistent => true
port => 5670
user => "USER"
password => "PASS"
#workers => 4
}

Activity

Show:
Will Appleby
March 27, 2014, 10:14 AM

I take it this is still unresolved as of Logstash 1.4? It's frustrating that we can't upgrade past 1.2.x because of this single issue, but using %{field} to dynamically set the key is integral to how we've set up our log shipping infrastructure so we can't upgrade until it's fixed.

Does anyone have plans to look at this? I'd love to sort it myself but I'm not a Ruby developer...

Will Appleby
July 9, 2014, 8:25 AM

Does anyone have an update on this or likelihood of it being fixed anytime soon? It's prevent us from upgrading past 1.2.2 on all our shipper nodes as we depend on the dynamic routing to individual rabbitmq queues for our log shipping architecture.

I'd love to fix it myself but I am not a Ruby dev so wouldn't really know where to start...

Philippe Weber
July 11, 2014, 4:33 AM

Sorry I'm not usin rabbitmq myself but looking at the code, it seems that the dynamic value is correctly handled in the ruby impl but not the jruby impl, default used by logstash package.
This is due to the refactoring to use codec in march_hare.

You could compare the two impl
https://github.com/elasticsearch/logstash/blob/master/lib/logstash/outputs/rabbitmq/bunny.rb
https://github.com/elasticsearch/logstash/blob/master/lib/logstash/outputs/rabbitmq/march_hare.rb

and do the following changes in march_hare.rb to try it out, supposing you publish json to rabbitmq

1. Add require "logstash/json" to the top of the file (just after the #encoding line)
2. Add the following line (taken from bunny.rb) to the register method here https://github.com/elasticsearch/logstash/blob/master/lib/logstash/outputs/rabbitmq/march_hare.rb#L29

3. Replace the line https://github.com/elasticsearch/logstash/blob/master/lib/logstash/outputs/rabbitmq/march_hare.rb#L31
replacing @codec.encode(event) by publish_serialized(event.to_json, key)
4. Change the method declaration here https://github.com/elasticsearch/logstash/blob/master/lib/logstash/outputs/rabbitmq/march_hare.rb#L38
from def publish_serialized(message) to def publish_serialized(message, key = @key)
5. Last change, on this https://github.com/elasticsearch/logstash/blob/master/lib/logstash/outputs/rabbitmq/march_hare.rb#L41
replace @key by simply key

This should be enough to restore dynamic key for you.
I will report the issue in technical term to github hoping logstash team can come up with a better fix than removing codec

Colin Surprenant
July 22, 2014, 8:30 PM

Thanks Philippe - we'll followup on this issue in https://github.com/elasticsearch/logstash/issues/1523. It's an easy fix, we should make that happen for 1.4.3. If you don't want to wait for the 1.4.3 release, once the fix is merged, you can ask for a nightly build.

Colin

Colin Surprenant
July 22, 2014, 8:31 PM

Assignee

Logstash Developers

Reporter

Thomas Spicer

Labels

None

Affects versions

Configure