Issues
- SQS Input - slow performanceLOGSTASH-1968Logstash Developers
- Performance issues with UDP inputLOGSTASH-1850Resolved issue: LOGSTASH-1850John Arnold
- Insufficient performanceLOGSTASH-1771Resolved issue: LOGSTASH-1771Jordan Sissel
- Grok fields type conversion is slowLOGSTASH-1500Resolved issue: LOGSTASH-1500Logstash Developers
- rabbitmq publish/delivery ratio still hurting after 1.2.1 upgradeLOGSTASH-1318Jason Kendall
- TCP input events are expensiveLOGSTASH-967Resolved issue: LOGSTASH-967Former user
- input redis batch pipeliningLOGSTASH-853Resolved issue: LOGSTASH-853Logstash Developers
- multiline filter has no resource limitsLOGSTASH-532Logstash Developers
8 of 8
Logstash SQS input performance seems to be very slow. The best numbers we observed from the Metrics filter output are:
~80-90 messages per second (use_ssl => true) - we do need to use SSL..
~200 messages per second (use_ssl => false)
I'd expect these numbers to be significantly higher...
Here's what I've tried to increase throughput:
Increased JVM heap (to 1G, 3G, 6G...)
Increased "threads" in SQS input configuration for Logstash (to 50, 100, 300, 500...)
Increased filter thread pool via the "-w" flag (to 20, 50, 100, 300, 500...)
Increased output thread pool (workers/threads settings) (to 20, 50, 100...)
Even tried to only have the SQS input defined (no filter, no output) - SQS queue counts seem to decrease at approximately the same rate
Environment:
m3.xlarge EC2 instance on AWS (64-bit, vCPU: 4, ECU: 13, 15 GB RAM, 2 x 40 SSD instance storage, Network Performance: High)
CentOS_6.3_x64_v5.8.8
JDK 1.7.0_51
Logstash: 1.3.3
Logstash agent command:
logstash.conf: