Duplicate Message sent?

Elastic Stack Logstash
1

Hi All,

there are 5 hosts (logstash-forwarder > redis > logstash-center > elasticsearch < Kibana), when I push an message to logstash-forwarder, redis always receives 2 messages, I try to delete the keys of list of redis, but it seems cannot work fine, how to solve the problem, following are my details setting

logstash-forwarder CONF
input {
file {
type => "type_count"
path => ["/data/logs/count/stdout.log", "/data/logs/count/stderr.log"]
exclude => ["*.gz", "access.log"]
}
}

output {
stdout { codec => rubydebug }
redis {
host => "redis"
port => 6379
data_type => "list"
key => "key_count_test"
}
}

[root@logstashagent ~]# echo "4999M 9 910 140.92.25.201" >> /data/logs/count/stdout.log

Redis
[root@redis ~]# redis-cli
127.0.0.1:6379> KEYS *
(empty list or set)

[root@redis ~]# redis-cli monitor
1436839399.032077 [0 140.92.25.88:37476] "blpop" "key_count_test" "0" "1"
1436839399.312106 [0 140.92.25.201:45951] "rpush" "key_count" "{"message":"4999M 9 910 140.92.25.201","@version":"1","@timestamp":"2015-07-14T02:02:28.251Z","host":["logstashagent","140.92.25.201"],"path":"/data/logs/count/stdout.log","type":"type_count","memory":"4999M","used":"9","avaliable":"910"}"
1436839399.315235 [0 140.92.25.88:37463] "blpop" "key_count" "0" "1"
1436839399.912829 [0 140.92.25.201:45958] "rpush" "key_count_test" "{"message":"4999M 9 910 140.92.25.201","@version":"1","@timestamp":"2015-07-14T02:02:28.857Z","host":"logstashagent","path":"/data/logs/count/stdout.log","type":"type_count"}"

logstash-center CONF
input {
redis {
host => "redis"
port => 6379
type => "redis-input"
data_type => "list"
key => "key_count_test"
}
}

output {
stdout { codec => rubydebug }
elasticsearch {
host => "elasticsearch"
codec => "json"
protocol => "http"
}
}

Kibana message

July 14th 2015, 10:02:28.857 message:4999M 9 910 140.92.25.201 @version:1 @timestamp:July 14th 2015, 10:02:28.857 host:logstashagent path:/data/logs/count/stdout.log type:type_count _id:AU6KTUQ9neUCreKPrDO- _type:type_count _index:logstash-2015.07.14
July 14th 2015, 10:02:28.251 message:4999M 9 910 140.92.25.201 @version:1 @timestamp:July 14th 2015, 10:02:28.251 host:logstashagent, 140.92.25.201 path:/data/logs/count/stdout.log type:type_count memory:4999M used:9 avaliable:910 _id:AU6KTUHmneUCreKPrDO9 _type:type_count _index:logstash-2015.07.14

2

Check what files you have in /etc/logstash/conf.d. I suspect you have an extra configuration file with a redis output so that each message is sent to Redis twice.

Note: logstash-forwarder is the name of a program separate from Logstash. The configuration snippet you've shown is for Logstash. To avoid confusion, don't say logstash-forwarder when you mean Logstash.

3

Hi Magnus Back,

Thanks for replying,

I checked my files of /etc/logstash/conf.d (Logstash) as following
[root@logstashagent conf.d]# pwd
/etc/logstash/conf.d
[root@logstashagent conf.d]# ll
total 4
-rw-r--r--. 1 root root 430 Jul 14 11:17 logstashagent.conf

As my conf file, the key 'key_count' is not exist in output/redis/key anymore, but it always appears in Redis monitor, like
1436867048.833035 [0 140.92.25.88:37479] "blpop" "key_count_test" "0" "1"
1436867049.133807 [0 140.92.25.88:37464] "blpop" "key_count" "0" "1"
1436867049.434578 [0 140.92.25.88:37463] "blpop" "key_count" "0" "1"
1436867049.835537 [0 140.92.25.88:37479] "blpop" "key_count_test" "0" "1"
1436867050.136416 [0 140.92.25.88:37464] "blpop" "key_count" "0" "1"
1436867050.437172 [0 140.92.25.88:37463] "blpop" "key_count" "0" "1"

I tried to restart all services but the problem still exist

4

Hi Magnus,

finally I delete xxx.conf.swp file, and it works normally

Thanks