Details
-
Type:
Bug/Feature
-
Status: New (View workflow)
-
Resolution: Unresolved
-
Affects Version/s: 1.2.2
-
Fix Version/s: None
-
Labels:
Description
My IIS logs directory has a total of 250~ sub directories at 2 levels and a total of about 10,000~ files coming up to 850mb. When I try to run log stash to just grok all of them and output any errors I start to see _grokparsefailure tags appear and go into my stdout. The message that comes with it is mangled, like it started reading the line not at the beginning. Below I have listed all of th information that should help you debug the issue.
An important note... if I make the path property of the file input more specific it seems like it works fine. The more unspecific I make it the more erratic it gets.
LogStash: 1.2.2
JRE: 1.7.0_45
OS: Windows Server 2008 R2
Ran with: C:\>java -jar logstash-1.2.2-flatjar.jar agent -f logstash.conf -vv -l logstash.log
logstash.conf
input {
file
}
filter {
if [type] == "iis" {
if [message] =~ /^#/ {
drop { }
}
grok {
match => ["path", "IISLogs/(:?%
)/\d+/u_ex\d+_%
{NOTSPACE:server}.log"]
}
grok {
match => ["message", "(?<event_time>\d
-\d
{2}-\d{2}\d
{2}:\d{2}:\d
{2}) %
{IP:host_ip}%
{URIPROTO:method}%
{NOTSPACE:uri_path}(?:-|%
{NOTSPACE:uri_query}) %
{NUMBER:port:int}(?:-|%
{WORD:username}) %
{IP:client_ip}%
{NOTSPACE:user_agent}(?:-|%
{NOTSPACE:referer}) (?:-|%
{NOTSPACE:uri_host}) %
{NUMBER:status:int}%
{NUMBER:sub_status}%
{NUMBER:win32_status}%
{NUMBER:bytes_sent:int}%
{NUMBER:bytes_received:int}%
{NUMBER:time_taken:int}"]
}
date
{ match => ["event_time", "yyyy-MM-dd HH:mm:ss"] remove_field => ["event_time"] } }
}
output {
if "_grokparsefailure" in [tags] {
stdout
}
}
log output:
{:timestamp=>"2013-11-10T22:13:46.546000-0600", :message=>"Received line", :path=>"X:/IISLogs/My.IIS.Web.Project/131101/u_ex13110109_SERVERNAME.log", :text=>"09:59:28 10.10.0.1 GET /status - 443 - 10.10.0.1 A-User/Agent - 10.10.0.1 200 0 0 362 114 0\r", :level=>:debug, :file=>"/Users/cmoran/Downloads/logstash-1.2.2-flatjar.jar!/logstash/inputs/file.rb", :line=>"129"}
The actual line from the log:
2013-11-01 09:59:28 10.10.0.1 GET /status - 443 - 10.10.0.1 A-User/Agent - 10.10.0.1 200 0 0 362 114 0
Gliffy Diagrams
Attachments
Issue links
- discovered while testing
-
LOGSTASH-1503 file input: lost several characters in the first line
- Need Information