This might seem obvious to some of you guys but it took me a while and I think this should be logged somewhere without having to configure log4j. So logstash installs, my config passes, debug looks good, strace looks good. Ports are listening with netstat. Service is running, I do echo "this is a test" | nc -u ipaddr port – and my service dies. No log of it anywhere. Well I was outputting to the ES and to a the file plugin for debug turns out because the service runs as the nologin user logstash it didn't have perms to write to that file, I found it by starting the service without sudo. chown fixed it. But it still drove me insane for a bit.

I want to stress that I had a working install, in production and a hard drive died, I rescued what I could but had to rebuild. I reimported my config to the new instance tweaked some stuff. It is my fault but it worries me that it didn't log it until I dropped to a different user.

after everything had given me the all clear
Errno::EACCES: Permission denied - /var/log/lsout.log
initialize at org/jruby/RubyFile.java:361
new at org/jruby/RubyIO.java:852
open at /opt/logstash/lib/logstash/outputs/file.rb:147
receive at /opt/logstash/lib/logstash/outputs/file.rb:64
handle at /opt/logstash/lib/logstash/outputs/base.rb:86
initialize at (eval):436
call at org/jruby/RubyProc.java:271
output at /opt/logstash/lib/logstash/pipeline.rb:266
outputworker at /opt/logstash/lib/logstash/pipeline.rb:225
start_outputs at /opt/logstash/lib/logstash/pipeline.rb:152