Duplicate records when scaling logstash

Elastic Stack Logstash
1

Hi,
I am using ELK stack like below

LSF==> logstash-Shipper-->Redis-->Indexer-->nGinx-->ES

I have scaled shipper and indexers as below

  1. CASE1 for all logs
    shipper.conf

    output {

       redis {
       	 host => "localhost"
        	 data_type => "list"
        	 key => "logstash"
        	 
       }
     	
     }

Indexer.conf

 input {
  redis {
    host => "localhost"
    data_type => "list"
    key => "logstash"
  
  }
}
  1. CASE2 for tomee access-logs

Shipper-url.conf

   output {

          redis {
          	 host => "localhost"
           	 data_type => "list"
           	 key => "access-logs"
           	 congestion_interval => 1
          	
          }

        }

indexer-url.conf

input {
  redis {
    host => "localhost"
    data_type => "list"
    key => "access-logs"
   
  }
}

I am expecting that indexers should pick up events with specific keys mentioned in input configuration.
However, its not exactly the result. There are duplicate logs I can see in Kibana.

Please tell me what is the wrong I am doing?

br,
Sunil Chaudhari

2

Are the shipper and indexer different instances?

3

Sorry for replying to an old post, but I would like to know how it finally worked for you.

4

Hi,
I reverted the configuration of 2 indexer instances, because I didn't find the root cause.
However, Later I found that there were multiple processes running behind. That might be the root cause. But I didn't tried it again.

br,
Sunil.

5

Thanks Sunil for the response.

I have shipper and indexer in one instance as @warkolm mentioned. I was testing out logstash with Redis and wrote 2 configuration files.
conf1 --> file input and Redis output, conf2 --> Redis input and Elasticsearch output.

I want to know why keeping multiple configuration files in /etc/logstash/conf directory duplicates events. Say, If I use different instances, one for indexer and another for shipper, this duplication doesn't happen. Also, if I ran logstash with just conf1 file and then ran logstash again with conf2 file, there was no duplication.

6

Have a look at the following threads: