Limit event buffers by total bytes in buffer, not just number of events
Description
I have noticed when running large amounts of log events through a SQS queue with buffering / batching switched on I get an error when Logstash tries to send packets of 270k or more. (There is a hard limit of 256K per packet with SQS)
I think the best way to do this would be to add the feature to stud::buffer and then have the SQS input plugin choose 256K as a default.
I would be up for implementing this myself but I'm finding trouble figuring out exactly how the buffer library code fits in to logstash. Doesn't seem to be in src/ or lib/ folders.
I have noticed when running large amounts of log events through a SQS queue with buffering / batching switched on I get an error when Logstash tries to send packets of 270k or more. (There is a hard limit of 256K per packet with SQS)
I think the best way to do this would be to add the feature to stud::buffer and then have the SQS input plugin choose 256K as a default.
I would be up for implementing this myself but I'm finding trouble figuring out exactly how the buffer library code fits in to logstash. Doesn't seem to be in src/ or lib/ folders.