I am attempting to filter through grok with named fields, then filter through a custom filter.
After grok when I call the custom filter the event seems to be the same as before grok.
I am confused on how to chain filters and maintain the changes as the event trickles through them.
I took the basic DNS filter and started with that.
The grok filter works, if I run it without the second filter for crowd I get the results I want in output.
Thanks Philippe, I spent an hour Friday after your comment, and again this morning.
I still seem to be misunderstanding the flow of data from logstash input -> filter (grok -> custom filter)
event before grok filter
event after grok filter
So at this point grok has successfully parsed the event
In my custom filter I have event.inspect and can see the data I want is in the event.
Above the field user1 or user2 would provide the info.
In my custom filter how do I reference these fields?
As told, just think of the event as an associative array.
Let's take 2 case
1. You decide that your filter will only lookup the field username, so you "hardcode"
2. You need the field name to be a parameter of the filter, so you do
and you invoke two instance of your filter
3 Or you change the filter to accept an array of value to convert now that you understand the flow
I got it working through a lot of trial and error. But it's still very confusing on how logstash handles an event when passed through the filter chain.
The documentation is not very clear; it's very high-level. Maybe I was looking for more details than needed.
Here is what I ended up doing: I don't like the solution, and will have to refactor.
I am not happy with this because I am under the impression I am supposed to create the rest-client object in the def register than update the params in the def filter as it's needed.
I wasn't able to get the rest-client params to work with GET, the Atlassian REST service uses ? parameters instead of /user/username/value type.
So when logstash is first started it loads the crowd filter into memory, but creates a new rest-client object every time it's called. This is my impression.... Is this what is happening?
Thanks for your help Philippe!
Yes you're right for the resource creation, but you should be able to put it as a field variable and declare it once in the register method
Have a (maybe second?) look to rest_client readme, https://github.com/rest-client/rest-client
you should be able to pass your param only when calling the get method
Should look more like this:
Thank you very much Philippe, your input has been very valuable.
I will document my experience and hopefully others will be able to learn from my confusion.
All is working now, I implemented your recommended changes.
It clicked in my head this morning that event["something"] is completely different than event[something] <--- without the surrounding double quotes.
I also got the params working for rest-client. I think my first failed attempt as shown at the beginning of this ticket was due to my lack of understanding how event[something] is different than event["something"].