Logstash -- bind - Address already in use - Error / Crash

Description

Also affects 1.3.2.

Was working for nearly a week, stopped to make a small change, when it fired back up it started giving me the error "bind - Address already in use". Reverted back to config which had been working, but not working now-- still giving the error. Something is apparently wrong but I don't know what. (I am fairly new to logstash). I compared the current conf file with the backup of the one that I know was "known-good-working" and they're byte-for-byte the same. None of the other config files have changed at all.

Can anybody help!?!?

Thanks!!

John

Here is more detail concerning my setup:

A Centos 6.5 Linux system with 8GB RAM and several TB hard disk configured to be a standalone central (syslog) log server.

I'm using the following software stack:

NXLOG 2.6.1131-1
LOGSTASH 1.3.3 (upgraded yesterday, was running 1.3.2 when problem started)
ELASTICSEARCH 0.90.7-1
KIBANA (version??-- whatever comes with Logstash)

Here is Logstash.conf file:

input {
tcp {
host => "localhost"
port => 5140
type => "json-tcp"
debug => true
}
}

filter {
mutate {
rename => ["Message", "message"]
}
if [type] == "syslog" {
grok {
match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
if !("_grokparsefailure" in [tags]) {
mutate {
replace => [ "@source_host", "%{syslog_hostname}" ]
replace => [ "@message", "%{syslog_message}" ]
}
}
mutate {
remove_field => [ "syslog_hostname", "syslog_message", "syslog_timestamp" ]
}
}
}

output {
stdout {
debug => true
}

elasticsearch {

  1. embedded => true
    host => "localhost"
    port => "9300"
    cluster => "mars-logger"
    node_name => "mars-logger-100"
    codec => json {
    charset => "UTF-8"
    }
    }
    }

Activity

Show:
John Whitten
January 29, 2014, 5:05 PM

This is now fixed / resolved.

I made a backup of the config file in the same directory as the actual config file. When Logstash runs, it apparently takes whatever it finds in the config directory, concats it all together and uses it as the config. When it found the backup, it attempted to start a new listener on the port, and found the one from the "other" config file was already there doing the job and bailed with the "bind - Address in Use" message. (Many thanks to @epel on the IRC channel for the help in figuring this out!) Solution was simple, move the backup config file out of the config directory. Also he pointed out that you can break up your config file across many multiple config files that way and make it easier to config just what you need.

John Whitten
January 29, 2014, 5:06 PM

This is now fixed / resolved.
I made a backup of the config file in the same directory as the actual config file. When Logstash runs, it apparently takes whatever it finds in the config directory, concats it all together and uses it as the config.

(See longer comment attached to issue)

Assignee

John Whitten

Reporter

John Whitten

Labels

None

Fix versions

Affects versions

Configure