Details

      Description

      My IIS logs directory has a total of 250~ sub directories at 2 levels and a total of about 10,000~ files coming up to 850mb. When I try to run log stash to just grok all of them and output any errors I start to see _grokparsefailure tags appear and go into my stdout. The message that comes with it is mangled, like it started reading the line not at the beginning. Below I have listed all of th information that should help you debug the issue.

      An important note... if I make the path property of the file input more specific it seems like it works fine. The more unspecific I make it the more erratic it gets.

      LogStash: 1.2.2
      JRE: 1.7.0_45
      OS: Windows Server 2008 R2
      Ran with: C:\>java -jar logstash-1.2.2-flatjar.jar agent -f logstash.conf -vv -l logstash.log

      logstash.conf
      input {
      file

      { type => "iis" path => "X:/IISLogs/*/*/*.log" debug => true }

      }

      filter {
      if [type] == "iis" {
      if [message] =~ /^#/ {
      drop { }
      }

      grok {
      match => ["path", "IISLogs/(:?%

      {HOST:application}

      )/\d+/u_ex\d+_%

      {NOTSPACE:server}

      .log"]
      }

      grok {
      match => ["message", "(?<event_time>\d

      {4}

      -\d

      {2}-\d{2}

      \d

      {2}:\d{2}

      :\d

      {2}

      ) %

      {IP:host_ip}

      %

      {URIPROTO:method}

      %

      {NOTSPACE:uri_path}

      (?:-|%

      {NOTSPACE:uri_query}

      ) %

      {NUMBER:port:int}

      (?:-|%

      {WORD:username}

      ) %

      {IP:client_ip}

      %

      {NOTSPACE:user_agent}

      (?:-|%

      {NOTSPACE:referer}

      ) (?:-|%

      {NOTSPACE:uri_host}

      ) %

      {NUMBER:status:int}

      %

      {NUMBER:sub_status}

      %

      {NUMBER:win32_status}

      %

      {NUMBER:bytes_sent:int}

      %

      {NUMBER:bytes_received:int}

      %

      {NUMBER:time_taken:int}

      "]
      }

      date

      { match => ["event_time", "yyyy-MM-dd HH:mm:ss"] remove_field => ["event_time"] }

      }
      }

      output {
      if "_grokparsefailure" in [tags] {
      stdout

      { codec => "rubydebug" }

      }
      }

      log output:
      {:timestamp=>"2013-11-10T22:13:46.546000-0600", :message=>"Received line", :path=>"X:/IISLogs/My.IIS.Web.Project/131101/u_ex13110109_SERVERNAME.log", :text=>"09:59:28 10.10.0.1 GET /status - 443 - 10.10.0.1 A-User/Agent - 10.10.0.1 200 0 0 362 114 0\r", :level=>:debug, :file=>"/Users/cmoran/Downloads/logstash-1.2.2-flatjar.jar!/logstash/inputs/file.rb", :line=>"129"}

      The actual line from the log:
      2013-11-01 09:59:28 10.10.0.1 GET /status - 443 - 10.10.0.1 A-User/Agent - 10.10.0.1 200 0 0 362 114 0

        Gliffy Diagrams

          Attachments

            Issue links

              Activity

                People

                • Assignee:
                  logstash-dev Logstash Developers
                  Reporter:
                  ChadMoran Chad Moran
                • Votes:
                  4 Vote for this issue
                  Watchers:
                  8 Start watching this issue

                  Dates

                  • Created:
                    Updated: