Extend File Plugin to support optional file paths
Description
Gliffy Diagrams
Activity
Show:
James Carpenter January 2, 2012 at 11:18 PM
I upgraded from logstash-1.1.0beta6-monolithic.jar to logstash-1.1.0beta7-monolithic.jar and successfully confirmed the fix.
Not only does does the logstash shipper not complain on start-up about missing paths, if/when the missing paths are subsequently created the shipper picks them up without requiring a restart of the shipper. This is true even if a path contains directories that don't initially exist.
Former user December 30, 2011 at 4:24 AM
This is already fixed in 1.1 beta. I re-wrote a bunch of the ruby filewatch module we use, and it's much smarter now:
behaves properly in this case (missing file is fine, picks it up if it shows up later)
handles globs as you described
keeps state on log file positions between restarts
Fixed
Details
Details
Assignee
Former user
Former user(Deactivated)Reporter
James Carpenter
James Carpenter(Deactivated)Labels
Fix versions
Affects versions
Created December 29, 2011 at 10:06 PM
Updated April 19, 2013 at 8:20 PM
Resolved December 30, 2011 at 4:24 AM
If the file plugin is given a path to a file that doesn't exist it currently throws an error. In many cases that is desirable, but in other deployment environments it would be much better if the file plugin would continually look for such files to eventually show up. I suggest adding a configuration flag "fail_nonexistant_paths" which defaults to true (current behavior).
========================
Background Information Motivating Request:
I am deploying logstash shipper agents across a large set of similar test environments. Each of these environments has a large number of nodes with a complex federation of services deployed on them. Many of the log files being monitored only exist once the relevant applications have been launched at least once. This is not the case when deploying to a newly configured environment.
Furthermore, some of environments have a smaller subset of services deployed than others. If a given service is deployed to a given node within a particular environment I know the exact name of the relevant log files to monitor. I just don't always know if the service has been deployed and executed when deploying a logstash shipper agent. Furthermore the permissions and ownership vary from file to file, so just touching each possible log file could cause other problems.
When deploying a shipper agent I would like to be able to say: "Please watch for these file paths. If they show up, please start shipping them. If not, don't freak out."