Limit event buffers by total bytes in buffer, not just number of events

Description

I have noticed when running large amounts of log events through a SQS queue with buffering / batching switched on I get an error when Logstash tries to send packets of 270k or more. (There is a hard limit of 256K per packet with SQS)

I think the best way to do this would be to add the feature to stud::buffer and then have the SQS input plugin choose 256K as a default.

I would be up for implementing this myself but I'm finding trouble figuring out exactly how the buffer library code fits in to logstash. Doesn't seem to be in src/ or lib/ folders.

Gliffy Diagrams

Activity

Show:

Details

Assignee

Reporter

Labels

Affects versions

Created October 28, 2013 at 8:13 AM
Updated March 25, 2015 at 5:38 AM