Multiline changing from a filter to a codec breaks my Cisco ACS config

Description

I run Logstash in a number of ways, one being a Syslog collector, receiving logs from a range of different (mostly Cisco) network devices; firewalls, routers, etc. and pertinently, a Cisco ACS which provides TACACS services.

The syslog format on the ACS is a bit "special" in that it takes the message that is normally logged to local storage, breaks it into n-number of 1K chunks and then sends each chunk prefixed with a traditional syslog prefix. Here's an example:

You can see the usual Syslog PRI field, timestamp, hostname, this CSCOacs_-prefixed facility, then an incrementing message ID, followed by the number of segments making up this message ID, and a 0-indexed segment ID, then a chunk of the original message payload. You can see the first chunk has its own (more precise) timestamp.

Up to and including 1.1.13 I was using the multiline filter like so to join the chunks together and create a single event:

At this point I send the event off to RabbitMQ which will get consumed and processed and indexed into Elasticsearch. I try and keep as little munging in this Syslog collecting configuration.

You can hopefully see I'm not really using the message ID or segment ID's, I just assume the first chunk always begins with a YYYY-MM-dd HH:mm:ss.SSS datestamp.

So anyway, now that multiline is a codec this configuration doesn't work. I don't think writing a dedicated codec will work as there needs to be some filtering first, plus the UDP input receives messages for all sorts of devices and it may be in the future I need to handle another device with its own special format so I'd need more than one codec.

The alternative I see is to write another multiline-ish filter that perhaps can just use the message ID and/or segment ID's to do the joining, I couldn't see any existing filter that might fit the bill. I'll take a look and see what I can come up with.

Yay for devices with weird log formats

Activity

Show:
Richard Pijnenburg
September 4, 2013, 1:01 PM

Ah okay, because you need to grok it first for the message part you need the multiline filter after the grok filter.

Nice discussion point.

Richard Pijnenburg
September 13, 2013, 11:16 AM

if im not mistaken with 1.2.1 the multiline filter came back.
Can you confirm its working as expected?

Cheers.

Matt Dainty
September 16, 2013, 11:57 AM

Ah yes, if the multiline filter is restored I guess this fixes it. If the multiline filter is not going to be removed again any time soon then there are some small enhancements I would like to make.

Richard Pijnenburg
September 16, 2013, 12:41 PM

I don't expect it to be going away any time soon as there is no solution for multiline filtering after grok for example.
Any enhancements are welcome

Fixed

Assignee

Logstash Developers

Reporter

Matt Dainty

Labels

None

Affects versions

Configure