exception":"java.lang.IllegalArgumentException: Invalid format

Description

Hello

I have a problem with date filter _ it doesn't work and I have somes log comming asynchronous...

MY CONF :

input {
amqp {
type => ""
host => "graystash"
port => "5673"
user => "logstash"
password => "pass4logstash"
#exchange => "fanout"
exchange => "rawlogs"
#name => "rawlogs"
vhost => "root"
}
}
filter {
grok {
type => "DHCP"
patterns_dir => "/opt/logstash/patterns.d/DHCP"
pattern => "%{DHCP}"
named_captures_only => false
}

date {
type => "DHCP"
timestamp => "MMM dd HH:mm:ss"
timestamp => "MMM d HH:mm:ss"
timestamp8601 => ISO8601
}
}

output {
stdout {
type => "DHCP"
debug => true
}

gelf {
type => "DHCP"
facility => "DHCP"
sender => "oban.in.karavel.com"
level => [ "%{severity}", "u" ]
host => "graystash.in.karavel.com"
port => 12201
}
}

MY PATTERNS:

  1.  

    1.  

      1.  

        1. DHCP ####

  2. Log Complete
    DHCP %{DHCPTimestamp:timestamp}\s%{DHCPServerIP}\s\w+\:\s%{DHCPStatus}\s(\w+\s|-\sIP\:\s)(%{DHCPClientMAC}\s(%{DHCPDescription})\svia\s%{DHCPInterface}|%{DHCPClientIP}(\s(from|to)\s%{DHCPClientMAC}\s(via\s%{DHCPInterface}|(%{DHCPDescription})\svia\s%{DHCPInterface})|,\sPORT\:\s%{DHCPSwitchPort},\sVLAN:\s%{DHCPVlan},\sSWITCH:%{DHCPSwitch}|\svia\s%{DHCPInterface}|\s(%{DHCPClientMAC})\svia\s%{DHCPInterface}|\sto\s%{DHCPClientMAC}\svia\s%{DHCPInterface}))

  1. Detail
    DHCPTimestamp (?:^\w{3}(\s|\ss)\d+\s\d+\:\d+\:\d+)
    DHCPServerIP (?:\d+\.\d+\.\d+\.\d+)
    DHCPStatus (?DHCPACK|DHCPINFORM|NEW LEASE|DHCPREQUEST|DHCPDISCOVER|DHCPOFFER))
    DHCPClientIP (?:\d+\.\d+\.\d+\.\d+)
    DHCPClientMAC (?:\w+\:\w+\:\w+\:\w+\:\w+\:\w+)
    DHCPSwitchPort (?:\d+\/\d+)
    DHCPInterface (?:.*)
    DHCPDescription (?:.+)
    DHCPVlan (?:\d+)
    DHCPSwitch (?:.*)

AND THE LOG:

WARN -- : Failed parsing date from field {"timestamp":"2012-02-16T11:10:48.772000 +0100","field":"timestamp","value":"Feb 02 06:54:06","exception":"java.lang.IllegalArgumentException: Invalid format: \"Feb 02 06:54:06\"","backtrace":["org/joda/time/format/DateTimeFormatter.java:683:in `parseDateTime'","file:/opt/logstash/logstash-1.1.0-monolithic.jar!/logstash/filters/date.rb:101:in `register'","org/jruby/RubyProc.java:258:in `call'","file:/opt/logstash/logstash-1.1.0-monolithic.jar!/logstash/filters/date.rb:149:in `filter'","org/jruby/RubyArray.java:1612:in `each'","file:/opt/logstash/logstash-1.1.0-monolithic.jar!/logstash/filters/date.rb:143:in `filter'","org/jruby/RubyArray.java:1612:in `each'","file:/opt/logstash/logstash-1.1.0-monolithic.jar!/logstash/filters/date.rb:136:in `filter'","org/jruby/RubyHash.java:1175:in `each'","file:/opt/logstash/logstash-1.1.0-monolithic.jar!/logstash/filters/date.rb:128:in `filter'","file:/opt/logstash/logstash-1.1.0-monolithic.jar!/logstash/filterworker.rb:57:in `filter'","org/jruby/RubyArray.java:1612:in `each'","file:/opt/logstash/logstash-1.1.0-monolithic.jar!/logstash/filterworker.rb:49:in `filter'","org/jruby/RubyArray.java:1612:in `each'","file:/opt/logstash/logstash-1.1.0-monolithic.jar!/logstash/filterworker.rb:48:in `filter'","file:/opt/logstash/logstash-1.1.0-monolithic.jar!/logstash/filterworker.rb:31:in `run'","file:/opt/logstash/logstash-1.1.0-monolithic.jar!/logstash/agent.rb:654:in `run_filter'","file:/opt/logstash/logstash-1.1.0-monolithic.jar!/logstash/agent.rb:393:in `run_with_config'"],"message":"Failed parsing date from field","file":"file","line":"/opt/logstash/logstash-1.1.0-monolithic.jar!/logstash/filters/date.rb","method":"186","level":"warn"}

-------------------------------

Real log timestamp : Feb 02 06:54:06
@timestamp : 2012-02-16T10:09:13.913000Z
gelf timestamp : 1329386953

I'm not an expert... but nothing seems good
Any suggestions ?

Thx
T.D

Activity

Show:
Thibault Desaules
February 16, 2012, 10:33 AM
Thibault Desaules
February 17, 2012, 5:26 PM

I make a test filter to parse the timestamp but its strange....

  1. Call this file 'foo.rb' (in logstash/filters, as above)
    require "logstash/filters/base"
    require "logstash/namespace"
    require "date"
    require "time"

class LogStash::Filters::Foo < LogStash::Filters::Base

  1. Setting the config_name here is required. This is how you

  2. configure this filter from your logstash config.
    #

  3. filter {

  4. foo { ... }

  5. }
    config_name "foo"
    plugin_status "stable"

  1. Replace the message with this value.
    config :message, :validate => :string

public
def register

  1. nothing to do
    end # def register

public
def filter(event)
if @message

  1. Replace the event message with our message as configured in the

  2. config file.

  3. If no message is specified, do nothing.
    d = DateTime.strptime(event.timestamp, '%b %d %H:%M:%S')
    event.timestamp = d.strftime("%Y-%m-%dT%H:%M:%S")
    end
    end # def filter
    end # class LogStash::Filters::Foo

That's work but graylog2 don't use timestamp gelf field I think _

Philippe Weber
June 22, 2012, 2:35 PM

HI Thibault,
The date filter is used to set the @timestamp value from a field, indeed so something must not be working for you
I cannot reproduce your issue with the info you give here so I doubt a little that everything is in sync
What I would suggest is to simplify the date filter to
date { type => "DHCP" timestamp => "MMM dd HH:mm:ss"}

You can make a quick test using this config and comment back,

Philippe Weber
November 22, 2012, 4:10 PM

Feel free to reopen if you need support again

Assignee

Logstash Developers

Reporter

Thibault Desaules

Labels

Affects versions

Configure