I've recently installed the latest version of Logstash (1.3.3) and have spent the last couple of days trying troubleshooting the follow issue. I am able to receive logs from my test lumberjack agent and can view them in their raw form in Kibana. I have used the grok debug tool, read the documentation and tried a number of different methods for structuring my grok block. However, even the simplest grok filters return _grokparsefailure. When I use the stdin input method on my central index server, grok functions as intended. When I use the stdin method on the remote server, I get _grokparsefailure for the exact same input. I have deployed agents on two different OS's (gentoo and Ubuntu) and observe the same behavior. Any insight into this issue or tips on how to perform more sophisticated debugging methods for logstash would be very much appreciated!
Central indexer conf file:
For what it's worth the apache logs are in this custom format shown below. However, I have been debugging my code primarily by trying to get grok to recognize the stdin input from the remote server.
"Feb 14 20:10:50 SevOne_5 apache2_access: 10.2.18.41 - - [14/Feb/2014:20:10:50 +0000] "POST /doms/alerts/titleBar.php?extendSession=0 HTTP/1.1" 200 88"