We're updating the issue view to help you get more done. 

Logdate field with format YYYY-MM-dd HH:mm:ss,SSS always converted to 2014-01-01T00:33:33.000Z

Description

When parsing Tomcat logs, logs with @fields.logdate that looks like 2013-07-16 14:49:48,932 are always parsed to 2014-01-01T00:33:33.000Z, regardless of the actual contents of @fields.logdate or the date/time logstash sees the event.

For these logs, @fields.logdate is parsed with this grok pattern:

1 (?<logdate>%{DATE} %{TIME})

And this is the date filter pattern intended for these logs:

1 YYYY-MM-dd HH:mm:ss,SSS

I have also tried using the following grok and date patterns (respectively) to ignore the fractions of a second, but still get the same behavior:

1 2 (?<logdate>%{DATE} %{TIME}),%{INT} YYYY-MM-dd HH:mm:ss

I have narrowed it down to an issue with the %{TIME} portion of the field. When using (?<logdate>%{DATE}) as the grok pattern and YYYY-MM-dd as the date pattern, @timestamp is converted to @fields.logdate correctly. The incorrect behavior occurs when any portion of %{TIME} is parsed.

But wait, it gets more interesting!

When I append the date pattern YYYY-MM-dd HH:mm:ss,SSS anywhere after "UNIX" in the date hash, @timestamp is incorrectly parsed as described above. If I put the date pattern BEFORE "UNIX" in the date hash, the @timestamp is converted correctly, but Elastic Search repeatedly fails to index the event, forcing me to restart logstash.

Environment

None

Status

Assignee

Philippe Weber

Reporter

akat

Affects versions

1.1.13

Priority