Logdate field with format YYYY-MM-dd HH:mm:ss,SSS always converted to 2014-01-01T00:33:33.000Z


When parsing Tomcat logs, logs with @fields.logdate that looks like 2013-07-16 14:49:48,932 are always parsed to 2014-01-01T00:33:33.000Z, regardless of the actual contents of @fields.logdate or the date/time logstash sees the event.

For these logs, @fields.logdate is parsed with this grok pattern:

And this is the date filter pattern intended for these logs:

I have also tried using the following grok and date patterns (respectively) to ignore the fractions of a second, but still get the same behavior:

I have narrowed it down to an issue with the %{TIME} portion of the field. When using (?<logdate>%{DATE}) as the grok pattern and YYYY-MM-dd as the date pattern, @timestamp is converted to @fields.logdate correctly. The incorrect behavior occurs when any portion of %{TIME} is parsed.

But wait, it gets more interesting!

When I append the date pattern YYYY-MM-dd HH:mm:ss,SSS anywhere after "UNIX" in the date hash, @timestamp is incorrectly parsed as described above. If I put the date pattern BEFORE "UNIX" in the date hash, the @timestamp is converted correctly, but Elastic Search repeatedly fails to index the event, forcing me to restart logstash.



Philippe Weber



Affects versions