We'd like to instrument logstash with stasd support so we can carefully monitor its performance. We would like to write our patch in a way that it could hopefully be merged upstream; any guidance you can give us would be very much appreciated!
Here are some of the things we'd like to accomplish:
1. For the http output plugin, send a timer to statsd when a message is successfully sent or if it fails.
2. For the http output plugin, send a gauge to statsd that reports the delta between the current time and the timestamp in the log message. This will tell us exactly how far behind logstash is.
3. For each of the input/filter/output queues we would like to send a gauge with the current number of events.
Any guidance you could give us on how to implement these instruments would be very much appreciated!