When defining new grok patterns, there are the following needs:
Tests, both positive and negative, are needed, to document/verify/fix corner cases.
For long-term maintainability, examples should be kept and re-run.
Examples should be near the pattern.
Logstash users are not necessarily programmers; writing Ruby tests is probably too much to ask.
Basically it's the same inputs/test data you'd have in a unit test, but stripped down to the bare essentials. Visually, the examples are indented under the pattern. This makes the ownership clear. "Yes/no" is a concise/readable signal for the upcoming test string.
Maybe comments would be a good thing? This is typical to describe test inputs, but complicates parsing...
Pattern files should be monitored and reloaded, on by default unless disabled for performance.
Saving a file parses the patterns, runs the rules, prints any failures. Upstream consumers of patterns should also be tested.
Long-term a GUI (a la Jenkins) could be envisioned that'd let the user edit patterns with fast feedback, but this solves the current usage pattern of editing files.
Regarding "Example should be near the pattern" I totally agree. I think we can resolve that separately with comments in the patterns files.
The code given in the url above (apache_example.rb) is an experiment. I take your point about logstash users not necessarily having ruby skills - I agree. I'll see what I can come up with.
Cool.. this seems quite granular though. 20 non-comment lines to check. Testing at a high-level of yes/no, having test data as rows, seems valuable. (Is that what you mean by " I think we can resolve that separately with comments in the patterns files."?) See, for example, the "where" construct in Spock – http://meetspock.appspot.com/edit/9001, click "run script".
I'm not clear - what's the use case for a non-programmer testing their patterns? Are they writing ruby code? What's the feedback loop? Because I could easily imagine a Web UI where they plop in a bunch of lines and it shows what lines match..what results are. If the GUI updated the tests, would be very powerful while serving programmers and non-programmers alike.
The problem with tests things appearing in the patterns file is that I'd have to invent syntax for describing these tests. Syntax which would require sufficient complexity as to do all the testing you might want to do in a grok pattern (verify a successful match, verify certain fields, verify field types, etc). On the flip-side, I think having more complex patterns having documentation-comments surrounding them seems reasonablem at least as a start.
The idea with the separate test suite that you can say "Here is my logstash config" and "here are some sample events" and finally "This is what each event should look like" - it should act as a more approachable way to test logstash configs. The particular interface isn't as important; as long as folks can repeatably test their configs against sample logs, this should help folks get things going when filters aren't behaving as expected (due to bugs/misconfiguration/etc)
Regarding 'non programmers' and ruby, Chef has had quite some success in this area (using ruby among non-programmers), so I am not worried much about this, and even then I'm not certain the long-term testing tool will be pure ruby, so am even less concerned about ruby in this case
Reopening - this feature request is slightly different from a more general-purpose testing suite.
The new logstash rspec suite allows more flexibility here (config + sample logs)