Uploaded image for project: 'logstash'
  1. LOGSTASH-2280

S3 configuration doesn't work when loaded via folder

    Details

    • Type: Bug/Feature
    • Status: New (View workflow)
    • Resolution: Unresolved
    • Affects versions: 1.4.0
    • Fix versions: None
    • Labels:

      Description

      I’m using the S3 input plugin for logstash - it won’t parse S3 logs when I -f a folder with the s3 config, but it will parse them when I pass -f the config file. My apache configuration works fine both ways. I delete the sincedb file each time I test, and am not too sure what to try next.

      I have the following structure:

      /etc/logstash/patterns/s3
      /etc/logstash/config/s3.conf

      This works:

      sudo /usr/bin/logstash/bin/logstash agent --verbose -f /etc/logstash/conf/s3.conf

      This doesn't:

      sudo /usr/bin/logstash/bin/logstash agent --verbose -f /etc/logstash/conf/

      If I swap apache.conf (my apache configuration file, two file inputs) with s3.conf, parsing apache logs works either way.

      An ELF file is created in /tmp/ (jffi1616993874545511618.tmp), instead of the s3.sincedb file I'm expecting when it isn't working.

      I'm using logstash 1.4.2 on Amazon Linux, with Java 1.6.0.

      input {
      	s3 {
      		backup_to_dir => "/tmp/logstashed/"
      		bucket => "special-logging-bucket"
      		delete => true
      		interval => 60
      		prefix => "logs/example.com/"
      		credentials => ["USER", "KEY"]
      		region_endpoint => "us-east-1"
      		sincedb_path => "/tmp/s3.sincedb"
      		tags => ["s3", "example-cdn", "caen"]
      		type => "example-test"
      	}
      }
      
      filter {
      	grok {
      		patterns_dir => ["/etc/logstash/patterns/"]
      		match => [ "message", "%{S3}" ]
      	}
      
      	# Drop anything that starts with logs
      	if [request] =~ /^\/logs\// {
      		drop {}
      	}
      
      	if ([s3_operation] != "REST.GET.OBJECT") {
      		drop {}
      	}
      
      	date {
      		match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
      	}
      
      	# remove example.com from path
      	mutate {
      		gsub => [ "request", "/example.com", "" ]
      	}
      
      	mutate {
      		add_field => {
      			"host" => "example.com"
      		}
      	}
      
      	geoip {
      		source => "clientip"
      	}
      }
      
      output {
      	redis {
      		host => "555.555.555.555"
      		data_type => "list"
      		key => "logstash"
      		codec => json
      	}
      }
      

      S3 patterns:

      BUCKET_OWNER %{USERNAME:s3_bucket_owner}
      BUCKET %{HOSTNAME:s3_bucket}
      REQUESTER %{NOTSPACE:s3_requester}
      REQUEST_ID %{USERNAME:s3_request_id}
      OPERATION [A-Z.-_]+
      KEY %{NOTSPACE:s3_key}
      REQUEST_URI "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})"
      AWS_ERROR_CODE %{NOTSPACE:s3_aws_error_code}
      BYTES_SENT (?:%{NUMBER:bytes}|-)
      OBJECT_SIZE (?:%{NUMBER:s3_object_size}|-)
      TOTAL_TIME (?:%{NUMBER:s3_total_transfer_time}|-)
      TURN_AROUND_TIME (?:%{NUMBER:s3_turn_around_time}|-)
      VERSION_ID %{NOTSPACE:s3_aws_version_id}
      
      S3 %{BUCKET_OWNER} %{BUCKET} \[%{HTTPDATE:timestamp}\] %{IP:clientip} %{REQUESTER} %{REQUEST_ID} %{OPERATION:s3_operation} %{KEY} %{REQUEST_URI} %{NUMBER:response} %{AWS_ERROR_CODE} %{BYTES_SENT} %{OBJECT_SIZE} %{TOTAL_TIME} %{TURN_AROUND_TIME} %{QS:referrer} %{QS:agent} %{VERSION_ID}
      

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                wiibaa Philippe Weber
                Reporter:
                ururk John P (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: