File input prevents log4j file rolling on Windows

Description

I'm using the simple, single standalone server of logstash on Windows 7 x64. I have a basic Java program writing log files using the log4j rolling file appender and I am monitoring it with logstash's file input. Below is my logstash.conf and log4j.xml from my program

logstash.conf:
input {
file {
type => "my-log"
path => [ "C:/logs/mylog/*.log" ]
}
}

output {
stdout { }
elasticsearch { embedded => true }
}

log4j.xml:
<?xml version="1.0" encoding="UTF-8" ?>
<log4j:configuration>
<appender name="FileAppender" class="org.apache.log4j.RollingFileAppender">
<param name="MaxBackupIndex" value="30"/>
<param name="MaxFileSize" value="1MB"/>
<param name="File" value="c:/logs/mylog/MyService.log"/>

<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d %-5p [%t] %c - %m%n"/>
</layout>
</appender>

<root>
<priority value="DEBUG" />
<appender-ref ref="FileAppender"/>
</root>
</log4j:configuration>

When logstash is monitoring the file it prevents the file from rolling. Once the file is full (1 MB in my example case for testing), the current .log file is truncated and it starts over. The other rolled files all roll as expected (the .1 file becomes .2, .2 becomes .3, etc.) leaving a hole where the .log file should have gone to .log.1. It is almost like logstash's open file handle is preventing the active .log file from being renamed.

If I stop logstash, the file will then roll just fine.

I have seen this before when another program other than the owning java process has an open file handle to the file. For example, if a developer is doing a tail on the .log file it will prevent it from rolling.

I did make a change to my logstash jar file per LOGSTASH-351. Since this modified the file handler reader, I'm wondering if that may have something to do with it. The problem is without making that change I get the error indicated in bug 351 and can't get logstash to pick up anything.

Activity

Show:
John E. Vincent
May 7, 2012, 3:34 PM

While I don't have an answer on the Windows issue, it might be more efficient where possible to use a custom appender instead of watching the file. I know that's not always an option but I've run the following two appenders at pretty high volume:

https://github.com/t0xa/gelfj
https://github.com/lusis/zmq-appender

Note that logstash 1.1 actually has support on the input side for both of those. I'm actively hacking on both logstash AND the zmq-appender.

The upshot of going to appender route is that you need to do less work on the logstash side with grok and ilk since everything comes in as JSON anyway. It's just a matter of selecting the appropriate keys.

I've not yet tested 0mq on Windows yet but the pyzmq team publish installers that include the appropriate DLLs - https://github.com/zeromq/pyzmq/downloads. My zmq-appender is only using 2.1 of zmq right now.

The gelf appender has no special external deps iirc.

Either way just shove the jar on the classpath, update your logger config and go.

Saravanan Bellan
May 11, 2012, 6:32 PM

Thanks for the suggestions. Since the volume of our events is very large we decided to send in compressed data to the log processing server. So we switched to the TimeandSizeRollingApppender from http://www.simonsite.org.uk/ which produces deterministic named files based on time and also controls file size and supports compression also. So we are creating an agent which moves these compressed files to log processing server which will then feed it to logstash. It just that our log processing could be delayed by an hour at most which is tolerable for our needs. So a suggested new input for logstash could be waiting for files to be created with specific time stamp and then send it to a another server where the agent will read these files and feed into elastisearch or other outputs. Not sure if any of the current inputs/outputs in logstash could accomplish that.

PS: I tried SplunkUniversalForwarder to forward raw data via tcp to logstash also. It seemed to work for a little bit but after a while I noticed that it also prevented the file from being rolling over.

Richard Pijnenburg
December 17, 2012, 4:53 AM

Very old ticket.
If issue still persists with current version please open a new ticket.

Boyd Meier
June 4, 2013, 5:47 PM

This is not resolved... WIll open a new ticket and link.

Joshua Chan
August 23, 2014, 4:24 AM

Has any progress been made on this?

Assignee

Logstash Developers

Reporter

D

Labels

None

Fix versions

Affects versions

Configure