# Logstash - Rabbitmq - same message repeatedly being loaded

**URL:** <https://discuss.elastic.co/t/logstash-rabbitmq-same-message-repeatedly-being-loaded/58533>\
**Category:** Logstash\
**Created:** [August 21, 2016, 2:26pm UTC](https://discuss.elastic.co/t/logstash-rabbitmq-same-message-repeatedly-being-loaded/58533 "2016-08-21T14:26:23Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![jaykumar](https://avatars.discourse-cdn.com/v4/letter/j/439d5e/32.png) [@jaykumar](https://discuss.elastic.co/u/jaykumar)\
**Post date:** [August 21, 2016, 2:26pm UTC](https://discuss.elastic.co/t/logstash-rabbitmq-same-message-repeatedly-being-loaded/58533/1 "2016-08-21T14:26:23Z")

</div>

Hi There,

I have configured rabbitmq as a broker for reliability, below is my configuration:

input {  
udp {  
port =\> 5114  
type =\> "syslog.imq"  
}  
}

output {

```
    if [type] == "syslog.imq" {

            rabbitmq {
                    key => "mykey"
                    exchange => "myex"
                    exchange_type => "direct"
                    user => "username"
                    password => "password"
                    host => "127.0.0.1"
                    port => 5672
                    durable => true
                    persistent => true
            }

    }

```

}

input {  
rabbitmq {  
host =\> "127.0.0.1"  
queue =\> "myq"  
durable =\> true  
key =\> "mykey"  
exchange =\> "myex"  
threads =\> 1  
prefetch\_count =\> 50  
port =\> 5672  
user =\> "user"  
password =\> "password"  
type =\> "syslog.raw"  
tags =\> "syslog.input"  
tags =\> "ti.pending"  
}  
}

filter {

```
    if "syslog.input" in [tags] {

 # formatting commands

                          mutate {

                                    remove_tag => "syslog.input"
                                    add_tag => "syslog.output"
                            }

}

```

}

output {

```
    if "syslog.output" in [tags] {

                    elasticsearch{

                            hosts => "localhost:9200"
                            action => "index"
                            index => "myfile-%{+YYYY.MM.dd}"
                            template => "/etc/logstash/file1.json"
                            template_name => "myfile-*"
                            manage_template => "true"
                            template_overwrite => "true"
                    }
            }
    }

```

Problem: One message is being repeatedly sent to Elasticsearch.

There are no errors in logstash and elasticsearch log files.

Please help me.

Regards,

Jay

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 21, 2016, 6:50pm UTC](https://discuss.elastic.co/t/logstash-rabbitmq-same-message-repeatedly-being-loaded/58533/2 "2016-08-21T18:50:18Z")

</div>

Is all of this in the same Logstash instance? Are you sure `type => "syslog.raw"` for the rabbitmq input overwrites the existing value?

---

<div class="post-metadata">

**Author:** ![jaykumar](https://avatars.discourse-cdn.com/v4/letter/j/439d5e/32.png) [@jaykumar](https://discuss.elastic.co/u/jaykumar)\
**Post date:** [August 22, 2016, 2:52am UTC](https://discuss.elastic.co/t/logstash-rabbitmq-same-message-repeatedly-being-loaded/58533/3 "2016-08-22T02:52:18Z")

</div>

Yes all of these are in same instance, I am really not sure how to ensure that Rabbitmq overwrites type = SYSLOG.IMQ. I would appreciate if you could give some hints.

I am building this to capture firewall SYSLOG events, using below workflow:

Firewall (TCP SYSLOG) -\> NXLOG (TCP SSL) -\> LS -\> RMQ -\> LS -\> ES

Not sure if I need two instances of LS (Logstash), I am trying to avoid multiple LS instances being Java base.

Alternatively I could use Fluentd instead of first instance of LS and then submit to RMQ -\> LS.

Please advise if there is any other solution.

Regards,

Ajay

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 22, 2016, 5:33am UTC](https://discuss.elastic.co/t/logstash-rabbitmq-same-message-repeatedly-being-loaded/58533/4 "2016-08-22T05:33:29Z")

</div>

> I am really not sure how to ensure that Rabbitmq overwrites type = SYSLOG.IMQ.

Look at the events in Elasticsearch. What's their type?

> Not sure if I need two instances of LS (Logstash), I am trying to avoid multiple LS instances being Java base.

Well, you either pay with RAM and CPU or with configuration complexity. Your pick.

---

<div class="post-metadata">

**Author:** ![jaykumar](https://avatars.discourse-cdn.com/v4/letter/j/439d5e/32.png) [@jaykumar](https://discuss.elastic.co/u/jaykumar)\
**Post date:** [August 24, 2016, 2:11pm UTC](https://discuss.elastic.co/t/logstash-rabbitmq-same-message-repeatedly-being-loaded/58533/5 "2016-08-24T14:11:03Z")

</div>

Thank you.  
I have created two instances of Logstash on same server to avoid any looping.  
This is working fine.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:41am UTC](https://discuss.elastic.co/t/logstash-rabbitmq-same-message-repeatedly-being-loaded/58533/6 "2017-07-06T04:41:52Z")

</div>