S3 input and large buckets

Description

S3 input is takeing a long time until the first logfile is processed:

Running it with

shows me that the bucket is used. As soon as I start logstash I see via tcpdump that there is a lot of traffic between the host and s3 going on.

Now that bucket has right now 4451 .gz files just in the root folder. Subfolders have even more files.

If I create now another bucket and put only one of the log files in it I can see that this logfile is more or less immediately downloaded and processed.

Assignee

Logstash Developers

Reporter

volker

Labels

Affects versions

Configure