ZeroMQ output needs a way to specify topics on pub/sub

Description

So in my simplification of the zmq inputs and outputs, I missed a use case that I thought I tested. Additionally, sub is actually BROKEN because of this.

When you do topics in PUBSUB for 0mq, it works like this:

PUB side needs to use more than send_string. It needs to send a multipart message with topic as the first part THEN the message body. This is done with ZMQ::SNDMORE. We can also use sendmsgs with an array of parts (topic + message body)

On the SUB side, we need to handle stripping the topic off the message and consuming only the body. This is normally done with more_parts? or we can use recvmsgs which gives us an array of parts.

The lingering question is how do we handle defining this in the config syntax. While sockopt => ["ZMQ::SUBSCRIBE", "topic.*"] works for SUB, it doesn't work for PUB. Ideally we'd want to create a new config option simply called {{topic.

Thoughts?

Activity

Show:
John E. Vincent
February 6, 2012, 10:12 PM

FYI, there's a proposed fix here:

https://github.com/avishai-ish-shalom/logstash/commit/6627635c20181a8d7448a64cc0e2b0b5eb0c5cc8

from Avishai but I'd like input on config syntax as well as flow control around SNDMORE

John E. Vincent
February 7, 2012, 2:49 PM

After discussing with Jordan, I've added a topic config entry and wired up both PUB and SUB side to do the right thing.

Assignee

John E. Vincent

Reporter

John E. Vincent

Labels

None

Fix versions

Affects versions

Configure