Decouple message format from transport

Description

I want to explore the possibility of further decoupling event format from from transport.

Original use case

Graylog2 supports AMQP as an input mechanism. The problem is that the messages need to be GELF or syslog formatted. Currently our approach is that the plugin does any custom formatting for the output. In the case of gelf output, we format a gelf message. Well technically we let the library format a gelf message. end pendant.

So now we're faced with a need to allow output to graylog2 over amqp. So do we create a new plugin called gelf_amqp or do we bloat the config syntax for gelf to encompass amqp options? Neither sounds amenable to me.

Additional use cases

Many other things that we support as outputs also have secondary transport for support. One example is Graphite. Who knows what else may come down the pipe? (no pun intended).

Use case with Filters

We have filters today that are essentially 'codecs' that act on specified fields in the event. JSON filter is an example. I also want a CSV filter. Both CSV and JSON are serialization formats.

If we properly implement codecs, we can have a 'codec' filter that lets you use any codec on any field in an event. This will let people log CSV over syslog, etc.

Implementation

The idea is to add an optional parameter to outputs/base.rb that specifies the event format. This way, there's no need to define a custom plugin for es river, graphite river, or graylog river. You simply add an additional attribute to an existing output and the event is formatted differently. Longer term (or even as a big refactor) we might now have lib/logstash/formats so that community contribution is much easier.

Downside? ElasticSearch river currently also does the autoconfiguration stuff for you. How to address that?

Alternate implementation

Instead of doing format as an attribute, flip everything on its head and make transport the attribute. If the transport provided matches a plugin, that plugins attributes are pulled into the existing one. I don't like this approach in the least as I've described it but it's worth mentioning.

Assignee

Jordan Sissel

Reporter

John E. Vincent

Labels

None

Fix versions

Affects versions

Configure