# Logstash and RabbitMQ input duplicate messages

**URL:** <https://discuss.elastic.co/t/logstash-and-rabbitmq-input-duplicate-messages/254957>\
**Category:** Logstash\
**Created:** [November 10, 2020, 5:57pm UTC](https://discuss.elastic.co/t/logstash-and-rabbitmq-input-duplicate-messages/254957 "2020-11-10T17:57:20Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![tomsozolins](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomsozolins/32/57007_2.png) [@tomsozolins](https://discuss.elastic.co/u/tomsozolins)\
**Post date:** [November 10, 2020, 5:57pm UTC](https://discuss.elastic.co/t/logstash-and-rabbitmq-input-duplicate-messages/254957/1 "2020-11-10T17:57:21Z")

</div>

Hello!  
I have annoying problem which i am trying to solve for like a week now and i have no idea where to look anymore...

ElasticStack 7.9.3 on 3 node cluster (CentOS 7, 3.10.0-1127.19.1.el7.x86\_64).  
RabbitMQ 3.8.9  
Erlang 23.1.2

I'm trying to send log messages from MinIO object storage to RabbitMQ and then from RabbitMQ to Logstash input. The problem is that messages are recieved correctly, but they are always indexed in Elasticsearch as two documents with the same content except document ID.

Duplication also happens if i do test message publish from RabbitMQ GUI to the defined queue, so i guess MinIO is not the problem. Also in RabbitMQ GUI i see that there is only one message recieved in exchange and only one message pushed to queue, so i guess RabbitMQ is working fine also?

Here is a sample configuration used for logstash:

```auto
input {
    rabbitmq {
        id => "logstash-1-bucketevents"
        host => "ipaddress:5672"
        user => "secret"
        password => "secretpassword"
        heartbeat => 30
        durable => false
        queue => "bucketevents"
    }
}

output {
  elasticsearch {
    hosts => ["https://node1:9200", "https://node2:9200", "https://node3:9200"]
    user => "elastic"
    password => "secretpassword"
  }
  stdout { codec => rubydebug }
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 10, 2020, 6:27pm UTC](https://discuss.elastic.co/t/logstash-and-rabbitmq-input-duplicate-messages/254957/2 "2020-11-10T18:27:18Z")

</div>

You get two copies of the events but you have three copies of logstash?

If so I would start by modifying the logstash configurations to always [add](https://discuss.elastic.co/t/logstash-to-logstash-lumberjack-host-details/253390/2) the hostname where logstash is running. Then review the events. Do the duplicates only come from one host?

That said, this really sounds like a RabbitMQ question. If you have a single queue then messages are load balanced across clients. If you have multiple queues and a fanout or topic then the same message will go to multiple queues and therefore multiple clients.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 10, 2020, 7:17pm UTC](https://discuss.elastic.co/t/logstash-and-rabbitmq-input-duplicate-messages/254957/4 "2020-11-10T19:17:09Z")

</div>

Are you pointing path.config at a directory? If so, is it possible you have a second output configured in another file? logstash.conf.bak and logstash.conf, for example. logstash will read every file in the directory and concatenate them.

---

<div class="post-metadata">

**Author:** ![tomsozolins](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomsozolins/32/57007_2.png) [@tomsozolins](https://discuss.elastic.co/u/tomsozolins)\
**Post date:** [November 10, 2020, 7:17pm UTC](https://discuss.elastic.co/t/logstash-and-rabbitmq-input-duplicate-messages/254957/5 "2020-11-10T19:17:15Z")

</div>

> [@tomsozolins](#):
>
> I have 3 logstash instances behind haproxy loadbalancer and pacemaker cluster.  
> MinIO is sending messages to Haproxy RabbitMQ VIP. RabbitMQ is configured in 3 node cluster with queue mirror. There is single exchange with MinIO and it is binded to single queue for logstash. I have configured logstash now for direct queue type and the problem is still happening.
> 
> I event tried to shutdown haproxy and PCS cluster and try to send only to single logstash instance directly and the duplicate problem is still happening.

I have 3 logstash instances behind haproxy loadbalancer and pacemaker cluster.  
MinIO is sending messages to Haproxy RabbitMQ VIP. RabbitMQ is configured in 3 node cluster with queue mirror. There is single exchange with MinIO and it is binded to single queue for logstash. I have configured logstash now for direct queue type and the problem is still happening.

I event tried to shutdown haproxy and PCS cluster and try to send only to single logstash instance directly and the duplicate problem is still happening.

---

<div class="post-metadata">

**Author:** ![tomsozolins](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomsozolins/32/57007_2.png) [@tomsozolins](https://discuss.elastic.co/u/tomsozolins)\
**Post date:** [November 10, 2020, 7:26pm UTC](https://discuss.elastic.co/t/logstash-and-rabbitmq-input-duplicate-messages/254957/6 "2020-11-10T19:26:28Z")

</div>

All 3 nodes have two config files in /logstash/conf.d/:

```auto
input {
    udp {
        host => "logstash1-ip"
        port => 5514
        type => syslog
    }

tcp {
        host => "logstash1-ip"
        port => 5514
        type => syslog
    }
}

output {
  elasticsearch {
    hosts => ["https://node1:9200", "https://node2:9200", "https://node3:9200"]
    user => "elastic"
    password => "secret"
  }
  stdout { codec => rubydebug }
}

```

```auto
input {
    rabbitmq {
        id => "logstash-1-bucketevents"
        host => "10.11.15.141:5672"
        user => "rabbitmq"
        password => "secret"
        heartbeat => 30
        durable => false
        queue => "bucketevents"
    }
}

output {
  elasticsearch {
    hosts => ["https://node1:9200", "https://node2:9200", "https://node3:9200"]
    user => "elastic"
    password => "secret"
  }
  stdout { codec => rubydebug }
}

```

---

<div class="post-metadata">

**Author:** ![tomsozolins](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomsozolins/32/57007_2.png) [@tomsozolins](https://discuss.elastic.co/u/tomsozolins)\
**Post date:** [November 10, 2020, 7:33pm UTC](https://discuss.elastic.co/t/logstash-and-rabbitmq-input-duplicate-messages/254957/7 "2020-11-10T19:33:20Z")

</div>

Well i removed output from the second file and i get only single message now 😄  
I guess i didn't understand how logstash works with multiple config files...

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 10, 2020, 7:48pm UTC](https://discuss.elastic.co/t/logstash-and-rabbitmq-input-duplicate-messages/254957/8 "2020-11-10T19:48:26Z")

</div>

It is a common misunderstanding. You can use pipelines.yml and point path.config to the individual files in there if you want multiple pipelines.

---

<div class="post-metadata">

**Author:** ![tomsozolins](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomsozolins/32/57007_2.png) [@tomsozolins](https://discuss.elastic.co/u/tomsozolins)\
**Post date:** [November 10, 2020, 7:50pm UTC](https://discuss.elastic.co/t/logstash-and-rabbitmq-input-duplicate-messages/254957/9 "2020-11-10T19:50:07Z")

</div>

Thank you very much! Will try to implement multiple pipelines for easier overview.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 8, 2020, 7:50pm UTC](https://discuss.elastic.co/t/logstash-and-rabbitmq-input-duplicate-messages/254957/10 "2020-12-08T19:50:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.