Inputs should be duplicatable to run more than one of the same plugin

Description

Many many many many..
many

many people report performance problems with the AMQP input plugin. I think it's Bunny, but frankly I haven't had the energy to stomach more AMQP debugging to figure out why or where the slowdown is.

However, folks are also reporting that you can increase throughput if you start with this:

And simply duplicate exactly the same amqp config, you get more performance:

While I don't think this should be necessary (the input should be fast enough on its own, what is really CPU-bound here?), in many cases it is a nice workaround.

Since I think we'll need lmany of these work arounds, we might as well make it a feature, and I think we'll find it a good feature.

Proposed example:

For plugins to support this, we should make a mixin 'LogStash:lugin::Threadable' that adds 'config :threads, ... whatever' and otherwise set it such that the Agent (or something external to the plugin) can see how many threads to create.

The effect, for now, will be to exactly replicate the plugin N times where 'threads => N'

Activity

Show:
Nick Ethier
June 8, 2012, 6:16 PM
PeteF
June 10, 2012, 3:46 PM

pull merged by jordan

Assignee

Logstash Developers

Reporter

Jordan Sissel

Labels

None

Fix versions

Affects versions

Configure