multiline with feature to limit certain patterns

Description

I'm looking for the ability to match multiline, events delimited by regex, and grepping out other undesirable regex. Some lines are 1 line, some are 2 lines and some are more.

irc log below for context.
==============
1:46
stasherslasher
the input is jboss error.log and what I'm seeing is logstash is holding everything until it hits a java exception - so there's lots of errors that begin with ^20 that aren't java exceptions (no "at org..blahblah") but those are not being written out to file until a java exception ^\s happens
1:46
is this expected?
and if so - why does it behave that way? is it because of the grep removing \s and it's buffering everything until grep has work to do?
1:49
whack
stasherslasher: oh, the '^20' 'next' probably merges everything starting with '^20'

1:49
whack
hmm
1:49
crankycoder left the room (quit: Remote host closed the connection).

1:50
whack
I wonder if we should add a new 'what' value for the multiline plugin

1:50
stasherslasher
hrm - that might make sense, but even the java exception starts with a 20, unless the grep taking effect causes the break/processing

1:50
whack
matching '^20' and saying something like 'what => "new"' to say "This match marks a new message

1:50
stasherslasher
ah yes that would be handy
1:50
whack
stasherslasher: well, think about what you're saying in the multiline

1:50
whack
"any line starting with ^20 belongs to the next line"
so 30 lines starting with "^20" all become the same event

1:50
stasherslasher
ah right
1:50
ken_barber entered the room.

1:51
stasherslasher
and then the ^\s is new - which gets thrown away
that makes perfect sense

1:51
whack
yeah
can you file a bug?
ask for 'what => new' in multiline

whack
actually wasn't your use case to multiline but only capture the first 2 lines?

1:55
stasherslasher
yep - well first 2 lines of a java exception, but there are also some legitimate errors that appear in error logs that have no exception (no ^\s)
I want it all, but just no ^\s lines period, delimited by ^20
the lines without a java exception sometimes are one liners

1:57
whack
oh I see, so you want multiline in general, but for stack traces only you want to limit it to 2 lines

1:57
stasherslasher
exactly

1:57
whack
hmm
file that as a feature request

1:58
stasherslasher
sure

1:58
whack
should be solvable

Assignee

Logstash Developers

Reporter

Derek Murphy

Labels

Fix versions

Affects versions

Configure