# Logstash - rabbitmq config to get multiple queues data to multiple elastic indeces

**URL:** <https://discuss.elastic.co/t/logstash-rabbitmq-config-to-get-multiple-queues-data-to-multiple-elastic-indeces/328520>\
**Category:** Logstash\
**Created:** [March 25, 2023, 2:09pm UTC](https://discuss.elastic.co/t/logstash-rabbitmq-config-to-get-multiple-queues-data-to-multiple-elastic-indeces/328520 "2023-03-25T14:09:12Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![qrshat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/qrshat/32/117776_2.png) [@qrshat](https://discuss.elastic.co/u/qrshat)\
**Post date:** [March 25, 2023, 2:09pm UTC](https://discuss.elastic.co/t/logstash-rabbitmq-config-to-get-multiple-queues-data-to-multiple-elastic-indeces/328520/1 "2023-03-25T14:09:12Z")

</div>

**scenario:**  
rabbitmq have different queues more than three. I want to make logstash configuration to get data from the rabbitmq queue and index this data to Elasticsearch. In order to that I have created logstash conf files for EVERY queue SEPARATELY with below configuration.

```auto
input {
  rabbitmq {
    id => "rabbitmyq_id0"
    # connect to rabbit
    host => "rabbitmq_hostname"
    port => 5672
    vhost => "/"
    user => "user"
    password => "pass"
   queue => "WebApiLog"
    ack => false
    durable => true
    exchange => "logstash_WebApiLog"
    exchange_type => "fanout"
    key => "WebApiLog"
    arguments => {
                         "x-queue-type" => "quorum"
                }
  }
}

output {
  elasticsearch {
    hosts => ["https://elastichostname:9200"]
   index => "webapi-logs-%{+YYYY.MM.dd}"
  }
}

```

When I run the logstash. All of the rabbit exchange get in place and logstash start to collect data from rabbitmq and index this data to elasticsearch.

**The problem:**  
Although I created separate config for every queues separately. When I check the created indices in elastic , I see that every indices has data for ALL queues, not just data of queue it created. I want to do every indices has data that belong to queue it is created for.

Can someone help me, how I can do that logstash configuration.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 25, 2023, 3:24pm UTC](https://discuss.elastic.co/t/logstash-rabbitmq-config-to-get-multiple-queues-data-to-multiple-elastic-indeces/328520/2 "2023-03-25T15:24:39Z")

</div>

> [@qrshat](#):
>
> Although I created separate config for every queues separately.

How you create that?

What does your `pipelines.yml` looks like?

---

<div class="post-metadata">

**Author:** ![qrshat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/qrshat/32/117776_2.png) [@qrshat](https://discuss.elastic.co/u/qrshat)\
**Post date:** [March 25, 2023, 5:22pm UTC](https://discuss.elastic.co/t/logstash-rabbitmq-config-to-get-multiple-queues-data-to-multiple-elastic-indeces/328520/3 "2023-03-25T17:22:35Z")

</div>

I didn't make any modification in pipelines.yml file. Shoul I do ?  
My config file is in conf.d file and content of yml file is below

```auto
- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 25, 2023, 5:36pm UTC](https://discuss.elastic.co/t/logstash-rabbitmq-config-to-get-multiple-queues-data-to-multiple-elastic-indeces/328520/4 "2023-03-25T17:36:00Z")

</div>

If path.config points to multiple files then they are concatenated into a single configuration. Events from all of the inputs are sent to all of the outputs. You need to use [pipelines.yml](https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html) to run each configuration in a separate pipeline.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 25, 2023, 6:13pm UTC](https://discuss.elastic.co/t/logstash-rabbitmq-config-to-get-multiple-queues-data-to-multiple-elastic-indeces/328520/5 "2023-03-25T18:13:46Z")

</div>

> [@qrshat](#):
>
> I didn't make any modification in pipelines.yml file. Shoul I do ?

If you want the pipelines to run independent from each other you need to edit `pipelines.yml` to do that.

The way it is configured now it will merge all the `.conf` files inside `/etc/logstash/conf.d/` and run as one big pipeline, all the inputs will be sent to all the outputs, which is the situation you described.

Check the linke documentation on the previous answer on how to configure multiple pipelines.

---

<div class="post-metadata">

**Author:** ![qrshat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/qrshat/32/117776_2.png) [@qrshat](https://discuss.elastic.co/u/qrshat)\
**Post date:** [March 25, 2023, 10:20pm UTC](https://discuss.elastic.co/t/logstash-rabbitmq-config-to-get-multiple-queues-data-to-multiple-elastic-indeces/328520/6 "2023-03-25T22:20:13Z")

</div>

Thanks @leandrojmp, @Badger. It works such as my expectations after editing yml file 🙏

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 22, 2023, 10:20pm UTC](https://discuss.elastic.co/t/logstash-rabbitmq-config-to-get-multiple-queues-data-to-multiple-elastic-indeces/328520/7 "2023-04-22T22:20:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.