Syslog

Hello
I'm trying to get the syslog messages from a server to another using logstash.
The server where the logs come from is 130.190.224.91 (server1) the server where ELK is installed is 130.190.250.82 (elkserver)
I'm using rsyslogd on server1, and added . @@elkserver:5514 to /etc/rsyslog.conf
Here's my logstahs conf from elkserver
input {
udp {
port => 25826 # 25826 matches port specified in collectd.conf
buffer_size => 1452 # 1452 is the default buffer size for Collectd
codec => collectd { } # specific Collectd codec to invoke
type => collectd
}
syslog {
port => 5514
type => syslog
host => "130.190.250.82"
}
}
output {
elasticsearch {
hosts => ["130.190.250.82:9200"]
}
}

Error in /var/log/logstah.log

{:timestamp=>"2015-12-23T10:25:04.654000+0100", :message=>"retrying failed action with response code: 503", :level=>:warn}

i tested telnet server1 -> elkserver on port 5514, it works

on elkserver : lsof -nPi :5514
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
rsyslogd 1487 root 9u IPv4 32035 0t0 TCP 130.190.250.82:41415->130.190.250.82:5514 (ESTABLISHED)
java 6713 logstash 18u IPv6 33189 0t0 UDP 130.190.250.82:5514
java 6713 logstash 37u IPv6 33190 0t0 TCP 130.190.250.82:5514 (LISTEN)
java 6713 logstash 42u IPv6 33805 0t0 TCP 130.190.250.82:5514->130.190.224.91:35523 (ESTABLISHED)
java 6713 logstash 43u IPv6 33215 0t0 TCP 130.190.250.82:5514->130.190.224.91:35524 (ESTABLISHED)
java 6713 logstash 44u IPv6 33368 0t0 TCP 130.190.250.82:5514->130.190.250.82:41415 (ESTABLISHED)

on server1 (130.190.224.91), command lsof -nPi :5514

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
rsyslogd 2639 root 8u IPv4 14331 0t0 TCP 130.190.224.91:35523->130.190.250.82:5514 (ESTABLISHED)
rsyslogd 2639 root 9u IPv4 14334 0t0 TCP 130.190.224.91:35524->130.190.250.82:5514 (ESTABLISHED)

What is the state of your Elasticsearch cluster? Does it work if you temporarily replace the Elasticsearch output with output to stdout?

health status index pri rep docs.count docs.deleted store.size pri.store.size
red open logstash-2015.12.23 5 1
yellow open logstash-2015.12.21 5 1 544320 0 68.9mb 68.9mb
yellow open logstash-2015.12.22 5 1 313936 0 42.3mb 42.3mb
yellow open logstash-2015.12.20 5 1 544320 0 68.8mb 68.8mb
yellow open .kibana 1 1 19 3 40.1kb 40.1kb
yellow open logstash-2015.12.19 5 1 544320 0 73mb 73mb
yellow open logstash-2015.12.18 5 1 797113 0 98mb 98mb
yellow open logstash-2015.12.17 5 1 1287360 0 139.8mb 139.8mb
yellow open logstash-2015.12.16 5 1 1070261 0 117.7mb 117.7mb
yellow open logstash-2015.12.15 5 1 1273753 0 136.1mb 136.1mb
yellow open logstash-2015.12.14 5 1 772573 0 82mb 82mb

It worked yesterday

Your current index is in red status, which means you can not index into it, which in turn will stop log stash processing. Is there anything in the Elasticsearch logs explaining why the index is red?

I found this

/var/log/elasticsearch/{hostname}.log

[2015-12-23 09:03:29,298][WARN ][index.translog ] [node01] [logstash-2015.12.22][3] failed to delete temp file /var/lib/elasticsearch/EC1/nodes/0/indices/logstash-2015.12.22/3/translog/translog-4754855291700189999.tlog