ISO8601 date format not matching %{TIMESTAMP_ISO8601}

Description

I am unable to use the timestamp8601 => ISO8601 define with my current log format.

Log example:
2011-09-01 22:49:26.387 INFO application: Connection state changed

My pattern for the time looks like this:
CFSERVERTIME_OTHER %{MONTHNUM:month}/%{MONTHDAY:day} %{TIME:time}
CFSERVERTIME %{TIMESTAMP_ISO8601:timestamp8601}|%{CFSERVERTIME_OTHER:timestamp}
And the rest (probably irrelevant, but here it is)
CFSERVER %{CFSERVERTIME} %{WORD:severity} %{DATA:application} %{GREEDYDATA:message}

I got a workaround in place by defining my date as such:
date {
type => "cfserver"
timestamp8601 => "yyyy-MM-dd HH:mm:ss.SSS"
timestamp => "MM/dd HH:mm:ss"
}

If you need more info, just ask. I'm in IRC.

I think ISO8601 should work for this format, so this is a potential bug.

discovered while testing

Gliffy Diagrams

Activity

Show:

Philippe Weber August 30, 2014 at 6:14 AM

PR Merged

Philippe Weber April 11, 2014 at 7:22 AM

September 18, 2013 at 3:50 PM

i have a similar failing format which is not recognized as ISO8601 when surrounded by other characters, even though the short timezone offset is explicitly allowed:
http://en.wikipedia.org/wiki/ISO_8601#Time_offsets_from_UTC
" the zone designator would be "+01:00", "+0100", or simply "+01"."

e.g.:
pattern: %{GREEDYDATA},%{TIMESTAMP_ISO8601:timestamp},%{GREEDYDATA}
"foo,2013-09-18T15:51:43:614+02:00,bar" works.
"foo,2013-09-18T15:51:43:614,bar" works.
"foo,2013-09-18T15:51:43:614+02,bar" does not work.

testable with http://grokdebug.herokuapp.com

Philippe Weber July 4, 2012 at 8:58 AM
Edited

In fact JODA is right, a space is not allowed in ISO, only 'T', see http://en.wikipedia.org/wiki/ISO_8601#Combined_date_and_time_representations

The issue is present for a long-time and remains in logging framework like log4j and logback, but considered minor...
see http://jira.qos.ch/browse/LOGBACK-262
So it may continue to spread the existing confusion.

Now, what to do in Logstash to ensure consistency between grok and date filter ?
I would go for splitting the grok pattern in two, one *real* ISO8601 with the T only, and another one with space and a *very explicit name*,
then from this name, adding a special case to the date filter would be a breeze.
But I have no clue for the name plus it would be a breaking change

Jordan Sissel September 9, 2011 at 12:31 AM

>> a = org.joda.time.format.ISODateTimeFormat.dateTimeParser >> a.parseDateTime("2011-09-04 12:00:33.4444").to_s NativeException: java.lang.IllegalArgumentException: Invalid format: "2011-09-04 12:00:33.4444" is malformed at " 12:00:33.4444"
Fixed

Details

Assignee

Reporter

Fix versions

Created September 8, 2011 at 5:45 PM
Updated August 30, 2014 at 6:14 AM
Resolved August 30, 2014 at 6:14 AM