Codecs - Build out framework

Description

Track progress of building out the framework for encode and decode plugins. All work will be done in the codecs branch.

Gliffy Diagrams

Activity

Show:

Nick Ethier June 16, 2013 at 4:09 AM

Jordan Sissel June 16, 2013 at 2:53 AM

SO EXCITING.

I'm marking this fixed as I have successfully been able to write my own codec (multiline).

WOot.

1.2 is shining up nicely!

Nick Ethier May 17, 2013 at 5:33 PM

Proposed solution:

A codec will be another logstash plugin, but it will not have its own config block. Instead inputs and outputs will have a 'codec' paramater similar to how the current 'format' parameter is today. Many codecs will exist and be available to users, this will allow all intputs/outputs to benifity from features like batching, compression, encryption, etc...

To implement a codec, you must subclass LogStash::Codecs::Base and override the encode and decode methods. Decode is called when an input has data it wishes to turn into an event. It can pass additional field to the codec that it wishes to include in the event. Encode is called on the output when an event is emitted. The output must define an on_event block that the codec can call when it has data to push through the output. Both inputs and outputs must call enable_codecs to initialize the codec before use.

A proof of concept implementation can be found in this branch:
https://github.com/logstash/logstash/tree/codec-hacking
Compared to master:
https://github.com/logstash/logstash/compare/codec-hacking

Fixed

Details

Assignee

Reporter

Labels

Fix versions

Created April 21, 2013 at 12:36 AM
Updated September 3, 2013 at 11:12 PM
Resolved June 16, 2013 at 2:53 AM