Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

File input support for compressed files

Description

It would be awesome if the file input could handle compressed files.

My workflow:

I have a folder filled with gzipped files and I want to automatically pull those files into logstash anytime a new file is created. This may be more work that just handling compressed files. It may also include reevaluating the glob more often and reading from the beginning of the file when new files are added.

testing discovered

Gliffy Diagrams

Activity

Show:

AjitK August 6, 2014 at 9:30 AM

Is this feature implemented? Or is there any work around to use this feature? May be, listen for zip files, netcat file from zip file and send it to a tcp port?

john Phan October 10, 2013 at 3:43 PM

Is this being built into the s3 input or can I assume from the conversation and the documentation that a separate codec is being made available called gzip?

As the online documentation seems to indicate that this is being separated out in the credentials section.

So in my example, logs provided by Cloudfront which are dumped into an S3 bucket are gziped. The top of http://logstash.net/docs/1.2.1/inputs/s3 documentation sugests that in 1.2.1 the files ending in .gz are all ready process and each line with in the files ending in .gz will be an event.

Is this not true?

Regards

John

Chakra Yadavalli October 4, 2013 at 4:21 PM

Eagerly looking for GZIP decompression support for file input. Is there a solution I could try?

ken branco September 24, 2013 at 1:01 AM

is anyone working on the gzip codec? I've been looking at the existing codecs as examples. It looks like the gzip file read just needs to be implemented in the decode method of the base codec.

I'd like to fork and contribute, but don't want to duplicate if someone is already working on the gzip codec.

ken branco September 20, 2013 at 3:27 PM

Hi, just wondering which way you are leaning on the gzip feature. There seems to be nice support for gzip in Ruby.
http://ruby-doc.org/stdlib-1.9.3/libdoc/zlib/rdoc/Zlib/GzipReader.html

Details

Assignee

Reporter

Labels

Affects versions

Created June 24, 2013 at 5:02 PM
Updated February 13, 2015 at 10:01 PM
Configure