We're updating the issue view to help you get more done. 

S3 configuration doesn't work when loaded via folder

Description

I’m using the S3 input plugin for logstash - it won’t parse S3 logs when I -f a folder with the s3 config, but it will parse them when I pass -f the config file. My apache configuration works fine both ways. I delete the sincedb file each time I test, and am not too sure what to try next.

I have the following structure:

/etc/logstash/patterns/s3
/etc/logstash/config/s3.conf

This works:

sudo /usr/bin/logstash/bin/logstash agent --verbose -f /etc/logstash/conf/s3.conf

This doesn't:

sudo /usr/bin/logstash/bin/logstash agent --verbose -f /etc/logstash/conf/

If I swap apache.conf (my apache configuration file, two file inputs) with s3.conf, parsing apache logs works either way.

An ELF file is created in /tmp/ (jffi1616993874545511618.tmp), instead of the s3.sincedb file I'm expecting when it isn't working.

I'm using logstash 1.4.2 on Amazon Linux, with Java 1.6.0.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 input { s3 { backup_to_dir => "/tmp/logstashed/" bucket => "special-logging-bucket" delete => true interval => 60 prefix => "logs/example.com/" credentials => ["USER", "KEY"] region_endpoint => "us-east-1" sincedb_path => "/tmp/s3.sincedb" tags => ["s3", "example-cdn", "caen"] type => "example-test" } } filter { grok { patterns_dir => ["/etc/logstash/patterns/"] match => [ "message", "%{S3}" ] } # Drop anything that starts with logs if [request] =~ /^\/logs\// { drop {} } if ([s3_operation] != "REST.GET.OBJECT") { drop {} } date { match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ] } # remove example.com from path mutate { gsub => [ "request", "/example.com", "" ] } mutate { add_field => { "host" => "example.com" } } geoip { source => "clientip" } } output { redis { host => "555.555.555.555" data_type => "list" key => "logstash" codec => json } }

S3 patterns:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 BUCKET_OWNER %{USERNAME:s3_bucket_owner} BUCKET %{HOSTNAME:s3_bucket} REQUESTER %{NOTSPACE:s3_requester} REQUEST_ID %{USERNAME:s3_request_id} OPERATION [A-Z.-_]+ KEY %{NOTSPACE:s3_key} REQUEST_URI "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" AWS_ERROR_CODE %{NOTSPACE:s3_aws_error_code} BYTES_SENT (?:%{NUMBER:bytes}|-) OBJECT_SIZE (?:%{NUMBER:s3_object_size}|-) TOTAL_TIME (?:%{NUMBER:s3_total_transfer_time}|-) TURN_AROUND_TIME (?:%{NUMBER:s3_turn_around_time}|-) VERSION_ID %{NOTSPACE:s3_aws_version_id} S3 %{BUCKET_OWNER} %{BUCKET} \[%{HTTPDATE:timestamp}\] %{IP:clientip} %{REQUESTER} %{REQUEST_ID} %{OPERATION:s3_operation} %{KEY} %{REQUEST_URI} %{NUMBER:response} %{AWS_ERROR_CODE} %{BYTES_SENT} %{OBJECT_SIZE} %{TOTAL_TIME} %{TURN_AROUND_TIME} %{QS:referrer} %{QS:agent} %{VERSION_ID}

Environment

None

Status

Assignee

Philippe Weber

Reporter

John P

Labels

Affects versions

1.4.0

Priority