Logstash - 1 event ends up 22 times in elasticsearch

[lueneburger (Dirk L.)]

Hi everyone,

got a real strange issue right now, haproxy is sending 1 event but logstash send it 22 times to elasticsearch.

i did only 1 change in the config (tested before in a test environment, without any problems)

normal config

if [logstash] == "haproxy_log" {
        grok {  id => match_haproxy_pattern
                match => {"message" => "%{HAPROXYHTTP}"}
                match => {"message" => "%{HAPROXYTCP}"}
                }

changed config

if [logstash] == "haproxy_log" {
        grok {  id => match_haproxy_pattern
               patterns_dir => ["/etc/logstash/patterns.d/"]
               match => {"message" => [ "%{HAPROXYHTTP}", "%{HAPROXYTCP}", "%{HAPROXYHTTP2}", "%{HAPROXYTCP2}" ]}
                }

the reason for the change was simple, i need to remove from some frontends the client_ip:port in the logs.

so i just changed the patterns and put both in it.

The question would be now:

did i miss something with the grok filter?
does the change put a big pressure on the performance? and that maybe could explain it?

also at the time when i did the restart, logs were going up to 70,000 instead of 2,000 per 5minute interval.

thanks in advance for any information