Help needed to parse nginx error.log
Description
Gliffy Diagrams
Activity
konvolut November 28, 2013 at 4:18 PM
The following works great:
%{DATA} %{WORD:webserver} %{HOST:myhost}\-%{WORD:class}\: (?<timestamp>%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER}: %{GREEDYDATA:errormessage}(?:, client: (?<client>%{IP}|%{HOSTNAME}))(?:, server: %{IPORHOST:server})(?:, request: %{QS:request})?(?:, host: %{QS:host})?(?:, referrer: \"%{URI:referrer})Philipp H November 28, 2013 at 9:36 AM
This works for me
grok {
match => {
"message" => [
"%{DATESTAMP:mydate} \[%{DATA:severity}\] (%{NUMBER:pid:int}#%{NUMBER}: \*%{NUMBER}|\*%{NUMBER}) %{GREEDYDATA:mymessage}",
"%{DATESTAMP:mydate} \[%{DATA:severity}\] %{GREEDYDATA:mymessage}",
"%{DATESTAMP:mydate} %{GREEDYDATA:mymessage}"
]
}
add_tag => ["nginx_error_pattern"]
}
if ("nginx_error_pattern" in [tags]) {
date {
match => [ "mydate", "yyyy/MM/dd HH:mm:ss" ]
add_tag => ["timestamp_changed"]
}
grok {
match => {
"mymessage" => [
"server: %{DATA:[request_server]},"
]
}
add_tag => ["asy_nginx_error_ext_server"]
}
grok {
match => {
"mymessage" => [
"host: \"%{IPORHOST:[request_host]}\""
]
}
add_tag => ["asy_nginx_error_ext_host"]
}
grok {
match => {
"mymessage" => [
"request: \"%{WORD:[request_method]} %{DATA:[request_uri]} HTTP/%{NUMBER:[request_version]:float}\""
]
}
add_tag => ["asy_nginx_error_ext_request"]
}
grok {
match => {
"mymessage" => [
"client: %{IPORHOST:[clientip]}",
"client %{IP:[clientip]} "
]
}
add_tag => ["asy_nginx_error_ext_client"]
}
grok {
match => {
"mymessage" => [
"referrer: \"%{DATA:[request_referrer]}\""
]
}
add_tag => ["asy_nginx_error_ext_referrer"]
}
mutate {
replace => [ "short_message", "%{mymessage}" ]
}
}Details
Assignee
Logstash DevelopersLogstash DevelopersReporter
konvolutkonvolut
Details
Details
Assignee
Logstash Developers
Logstash DevelopersReporter
konvolut
konvolutCreated November 27, 2013 at 10:52 AM
Updated November 28, 2013 at 4:18 PM
Maybe there is anybody who can help me to find out the right pattern to parse nginx error.log with grok.
Message:
Nov 26 09:41:22 webserver8 mydomain.com-error: 2013/11/26 09:41:13 [warn] 2787#0: *39595376 a client request body is buffered to a temporary file /var/lib/nginx/body/0000251155, client: 127.0.0.1, server: myserver.com, request: "POST /some_request.php HTTP/1.1", host: "www.mydomain.com", referrer: "https://www.mydomain.com/advanced_search_result.php?keywords=someword"My opinion:
"%{DATESTAMP:timestamp} %{WORD:webserver} %{HOST:host}\-%{WORD:class}\: (?<timestamp>%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME}) [%{LOGLEVEL:severity}] %{POSINT:pid}#%{NUMBER}: %{GREEDYDATA:errormessage}(?:, client: %{IPORHOST:client}) (?:, server: %{IPORHOST:server}) "(?:, request: %{QS:request})?(?:, host: %{QS:host})? "(?:%{URI:referrer})"But it still doesnt work.
Does anyone have any advice for me? Thanks in advance1