filter chaining question

Description

I am attempting to filter through grok with named fields, then filter through a custom filter.

After grok when I call the custom filter the event seems to be the same as before grok.

I am confused on how to chain filters and maintain the changes as the event trickles through them.

I took the basic DNS filter and started with that.

The grok filter works, if I run it without the second filter for crowd I get the results I want in output.

Gliffy Diagrams

Activity

Show:

Zachary Buckholz July 8, 2014 at 4:45 PM

Thank you very much Philippe, your input has been very valuable.

I will document my experience and hopefully others will be able to learn from my confusion.

All is working now, I implemented your recommended changes.

It clicked in my head this morning that event["something"] is completely different than event[something] <--- without the surrounding double quotes.

I also got the params working for rest-client. I think my first failed attempt as shown at the beginning of this ticket was due to my lack of understanding how event[something] is different than event["something"].

Thanks again!

Philippe Weber July 8, 2014 at 4:48 AM

Yes you're right for the resource creation, but you should be able to put it as a field variable and declare it once in the register method
Have a (maybe second?) look to rest_client readme, https://github.com/rest-client/rest-client
you should be able to pass your param only when calling the get method

Should look more like this:

Zachary Buckholz July 7, 2014 at 10:21 PM

I got it working through a lot of trial and error. But it's still very confusing on how logstash handles an event when passed through the filter chain.

The documentation is not very clear; it's very high-level. Maybe I was looking for more details than needed.

Here is what I ended up doing: I don't like the solution, and will have to refactor.

logstash.conf

crowd.rb

I am not happy with this because I am under the impression I am supposed to create the rest-client object in the def register than update the params in the def filter as it's needed.

I wasn't able to get the rest-client params to work with GET, the Atlassian REST service uses ? parameters instead of /user/username/value type.

So when logstash is first started it loads the crowd filter into memory, but creates a new rest-client object every time it's called. This is my impression.... Is this what is happening?

Thanks for your help Philippe!

Philippe Weber July 4, 2014 at 7:18 PM

As told, just think of the event as an associative array.
Let's take 2 case

1. You decide that your filter will only lookup the field username, so you "hardcode"

2. You need the field name to be a parameter of the filter, so you do

and you invoke two instance of your filter

3 Or you change the filter to accept an array of value to convert now that you understand the flow

Zachary Buckholz July 4, 2014 at 6:04 PM

Thanks Philippe, I spent an hour Friday after your comment, and again this morning.

I still seem to be misunderstanding the flow of data from logstash input -> filter (grok -> custom filter)

event before grok filter

event after grok filter

So at this point grok has successfully parsed the event

logstash conf

In my custom filter I have event.inspect and can see the data I want is in the event.

event.inspect output

Above the field user1 or user2 would provide the info.

In my custom filter how do I reference these fields?

crowd.rb filter

Details

Assignee

Reporter

Created July 3, 2014 at 7:12 PM
Updated July 8, 2014 at 4:48 PM