Allow Examples as tests in grok patterns

Description

When defining new grok patterns, there are the following needs:

  • Tests, both positive and negative, are needed, to document/verify/fix corner cases.

  • For long-term maintainability, examples should be kept and re-run.

  • Examples should be near the pattern.

  • Logstash users are not necessarily programmers; writing Ruby tests is probably too much to ask.

Example:

Basically it's the same inputs/test data you'd have in a unit test, but stripped down to the bare essentials. Visually, the examples are indented under the pattern. This makes the ownership clear. "Yes/no" is a concise/readable signal for the upcoming test string.

Maybe comments would be a good thing? This is typical to describe test inputs, but complicates parsing...

Feedback/REPL loop
Pattern files should be monitored and reloaded, on by default unless disabled for performance.
Saving a file parses the patterns, runs the rules, prints any failures. Upstream consumers of patterns should also be tested.

Long-term a GUI (a la Jenkins) could be envisioned that'd let the user edit patterns with fast feedback, but this solves the current usage pattern of editing files.

Activity

Show:
Jordan Sissel
August 30, 2012, 7:33 AM

Regarding "Example should be near the pattern" I totally agree. I think we can resolve that separately with comments in the patterns files.

The code given in the url above (apache_example.rb) is an experiment. I take your point about logstash users not necessarily having ruby skills - I agree. I'll see what I can come up with.

Jeff Winkler
August 30, 2012, 9:13 AM

Cool.. this seems quite granular though. 20 non-comment lines to check. Testing at a high-level of yes/no, having test data as rows, seems valuable. (Is that what you mean by " I think we can resolve that separately with comments in the patterns files."?) See, for example, the "where" construct in Spock – http://meetspock.appspot.com/edit/9001, click "run script".

I'm not clear - what's the use case for a non-programmer testing their patterns? Are they writing ruby code? What's the feedback loop? Because I could easily imagine a Web UI where they plop in a bunch of lines and it shows what lines match..what results are. If the GUI updated the tests, would be very powerful while serving programmers and non-programmers alike.

Jordan Sissel
August 30, 2012, 4:14 PM

The problem with tests things appearing in the patterns file is that I'd have to invent syntax for describing these tests. Syntax which would require sufficient complexity as to do all the testing you might want to do in a grok pattern (verify a successful match, verify certain fields, verify field types, etc). On the flip-side, I think having more complex patterns having documentation-comments surrounding them seems reasonablem at least as a start.

The idea with the separate test suite that you can say "Here is my logstash config" and "here are some sample events" and finally "This is what each event should look like" - it should act as a more approachable way to test logstash configs. The particular interface isn't as important; as long as folks can repeatably test their configs against sample logs, this should help folks get things going when filters aren't behaving as expected (due to bugs/misconfiguration/etc)

Regarding 'non programmers' and ruby, Chef has had quite some success in this area (using ruby among non-programmers), so I am not worried much about this, and even then I'm not certain the long-term testing tool will be pure ruby, so am even less concerned about ruby in this case

Jordan Sissel
August 30, 2012, 4:25 PM

Reopening - this feature request is slightly different from a more general-purpose testing suite.

Jordan Sissel
April 19, 2013, 6:03 AM

The new logstash rspec suite allows more flexibility here (config + sample logs)

Assignee

Jordan Sissel

Reporter

Jeff Winkler

Labels

Fix versions

Configure