ISO8601 date format not matching %{TIMESTAMP_ISO8601}
Description
discovered while testing
Gliffy Diagrams
Activity
Philippe Weber August 30, 2014 at 6:14 AM
PR Merged
Philippe Weber April 11, 2014 at 7:22 AM
Submitted PR: https://github.com/elasticsearch/logstash/pull/769
O September 18, 2013 at 3:50 PM
i have a similar failing format which is not recognized as ISO8601 when surrounded by other characters, even though the short timezone offset is explicitly allowed:
http://en.wikipedia.org/wiki/ISO_8601#Time_offsets_from_UTC
" the zone designator would be "+01:00", "+0100", or simply "+01"."
e.g.:
pattern: %{GREEDYDATA},%{TIMESTAMP_ISO8601:timestamp},%{GREEDYDATA}
"foo,2013-09-18T15:51:43:614+02:00,bar" works.
"foo,2013-09-18T15:51:43:614,bar" works.
"foo,2013-09-18T15:51:43:614+02,bar" does not work.
testable with http://grokdebug.herokuapp.com
Philippe Weber July 4, 2012 at 8:58 AMEdited
In fact JODA is right, a space is not allowed in ISO, only 'T', see http://en.wikipedia.org/wiki/ISO_8601#Combined_date_and_time_representations
The issue is present for a long-time and remains in logging framework like log4j and logback, but considered minor...
see http://jira.qos.ch/browse/LOGBACK-262
So it may continue to spread the existing confusion.
Now, what to do in Logstash to ensure consistency between grok and date filter ?
I would go for splitting the grok pattern in two, one *real* ISO8601 with the T only, and another one with space and a *very explicit name*,
then from this name, adding a special case to the date filter would be a breeze.
But I have no clue for the name plus it would be a breaking change
Jordan Sissel September 9, 2011 at 12:31 AM
>> a = org.joda.time.format.ISODateTimeFormat.dateTimeParser
>> a.parseDateTime("2011-09-04 12:00:33.4444").to_s
NativeException: java.lang.IllegalArgumentException: Invalid format: "2011-09-04 12:00:33.4444" is malformed at " 12:00:33.4444"Details
Details
Assignee
Philippe WeberReporter
Greg Dallavalle
I am unable to use the timestamp8601 => ISO8601 define with my current log format.
Log example:
2011-09-01 22:49:26.387 INFO application: Connection state changed
My pattern for the time looks like this:
CFSERVERTIME_OTHER %{MONTHNUM:month}/%{MONTHDAY:day} %{TIME:time}
CFSERVERTIME %{TIMESTAMP_ISO8601:timestamp8601}|%{CFSERVERTIME_OTHER:timestamp}
And the rest (probably irrelevant, but here it is)
CFSERVER %{CFSERVERTIME} %{WORD:severity} %{DATA:application} %{GREEDYDATA:message}
I got a workaround in place by defining my date as such:
date {
type => "cfserver"
timestamp8601 => "yyyy-MM-dd HH:mm:ss.SSS"
timestamp => "MM/dd HH:mm:ss"
}
If you need more info, just ask. I'm in IRC.
I think ISO8601 should work for this format, so this is a potential bug.