filter chaining question
Description
Gliffy Diagrams
Activity

Zachary Buckholz July 8, 2014 at 4:45 PM
Thank you very much Philippe, your input has been very valuable.
I will document my experience and hopefully others will be able to learn from my confusion.
All is working now, I implemented your recommended changes.
It clicked in my head this morning that event["something"] is completely different than event[something] <--- without the surrounding double quotes.
I also got the params working for rest-client. I think my first failed attempt as shown at the beginning of this ticket was due to my lack of understanding how event[something] is different than event["something"].
Thanks again!

Philippe Weber July 8, 2014 at 4:48 AM
Yes you're right for the resource creation, but you should be able to put it as a field variable and declare it once in the register method
Have a (maybe second?) look to rest_client readme, https://github.com/rest-client/rest-client
you should be able to pass your param only when calling the get method
Should look more like this:

Zachary Buckholz July 7, 2014 at 10:21 PM
I got it working through a lot of trial and error. But it's still very confusing on how logstash handles an event when passed through the filter chain.
The documentation is not very clear; it's very high-level. Maybe I was looking for more details than needed.
Here is what I ended up doing: I don't like the solution, and will have to refactor.
logstash.conf
crowd.rb
I am not happy with this because I am under the impression I am supposed to create the rest-client object in the def register than update the params in the def filter as it's needed.
I wasn't able to get the rest-client params to work with GET, the Atlassian REST service uses ? parameters instead of /user/username/value type.
So when logstash is first started it loads the crowd filter into memory, but creates a new rest-client object every time it's called. This is my impression.... Is this what is happening?
Thanks for your help Philippe!

Philippe Weber July 4, 2014 at 7:18 PM
As told, just think of the event as an associative array.
Let's take 2 case
1. You decide that your filter will only lookup the field username, so you "hardcode"
2. You need the field name to be a parameter of the filter, so you do
and you invoke two instance of your filter
3 Or you change the filter to accept an array of value to convert now that you understand the flow

Zachary Buckholz July 4, 2014 at 6:04 PM
Thanks Philippe, I spent an hour Friday after your comment, and again this morning.
I still seem to be misunderstanding the flow of data from logstash input -> filter (grok -> custom filter)
event before grok filter
event after grok filter
So at this point grok has successfully parsed the event
logstash conf
In my custom filter I have event.inspect and can see the data I want is in the event.
event.inspect output
Above the field user1 or user2 would provide the info.
In my custom filter how do I reference these fields?
crowd.rb filter
I am attempting to filter through grok with named fields, then filter through a custom filter.
After grok when I call the custom filter the event seems to be the same as before grok.
I am confused on how to chain filters and maintain the changes as the event trickles through them.
I took the basic DNS filter and started with that.
The grok filter works, if I run it without the second filter for crowd I get the results I want in output.