# Two logstash http\_poller (API query) configs are outputting documents to each other's indices

**URL:** <https://discuss.elastic.co/t/two-logstash-http-poller-api-query-configs-are-outputting-documents-to-each-others-indices/365066>\
**Category:** Logstash\
**Created:** [August 17, 2024, 2:16pm UTC](https://discuss.elastic.co/t/two-logstash-http-poller-api-query-configs-are-outputting-documents-to-each-others-indices/365066 "2024-08-17T14:16:19Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![taniumalloy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taniumalloy/32/136858_2.png) [@taniumalloy](https://discuss.elastic.co/u/taniumalloy)\
**Post date:** [August 17, 2024, 2:16pm UTC](https://discuss.elastic.co/t/two-logstash-http-poller-api-query-configs-are-outputting-documents-to-each-others-indices/365066/1 "2024-08-17T14:16:19Z")

</div>

I have a unique issue in Logstash where my data is not being routed to the correct index name appropriatly. I have two conf.d config files that each use http\_poller as input source and are gathering data from seperate APIs.

Here are my config files

User API - 02-users-api.conf

```auto
input {
  http_poller {
    urls => {
	test => {
        # Supports all options supported by ruby's Manticore HTTP client
        method => get
        url => "https://jsonplaceholder.typicode.com/users"
        headers => {
          Accept => "application/json"
        }
     }
    }
    request_timeout => 60
    # Supports "cron", "every", "at" and "in" schedules by rufus scheduler
    # schedule => { cron => "* * * * * UTC"}
    schedule => { every => "1m" }
    codec => "json"
    # A hash of request metadata info (timing, response headers, etc.) will be sent here
    metadata_target => "http_poller_metadata"
  }
}
filter {
  mutate {
    remove_field => ["http_poller_metadata"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9200"] # Replace with your Elasticsearch host and port
    index => "users-01" # Replace with your desired index name
  }
 #stdout { codec => rubydebug}
}

```

Here is my second conf.d config file - 01-weather-api.conf

```auto
input {
  http_poller {
    urls => {
	test3 => {
        # Supports all options supported by ruby's Manticore HTTP client
        method => get
        url => "https://api.openweathermap.org/data/2.5/weather"
        headers => {
          Accept => "application/json"
        }
	query => {
          lat => "49"
          lon => "81"
          appid => "<API KEY HERE>"
        }
     }
    }
    request_timeout => 60
    # Supports "cron", "every", "at" and "in" schedules by rufus scheduler
    # schedule => { cron => "* * * * * UTC"}
    schedule => { every => "1m" }
    codec => "json"
    # A hash of request metadata info (timing, response headers, etc.) will be sent here
    metadata_target => "http_poller_metadata"
  }
}
filter {
  mutate {
    remove_field => ["http_poller_metadata"]
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9200"] # Replace with your Elasticsearch host and port
    index => "weather-01" # Replace with your desired index name
  }
  #stdout { codec => rubyrebug }
}

```

I am getting data from each, but the data from each source is going to each index... like this:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/8/48ccba99787e369b8d9a42eb1b46925d6978a334.png)

I dont understand how this is happening because my outputs for each conf.d config file specify the unique index name that should be associated with the API.

When I test each conf.d file at a time, it works as expected, but when they are both active, I get this weird duplication and mixing of data in both indices.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [August 17, 2024, 2:37pm UTC](https://discuss.elastic.co/t/two-logstash-http-poller-api-query-configs-are-outputting-documents-to-each-others-indices/365066/2 "2024-08-17T14:37:26Z")

</div>

> [@taniumalloy](#):
>
> I dont understand how this is happening because my outputs for each conf.d config file specify the unique index name that should be associated with the API.

This is the default behaviour, files inside `conf.d` will be merged in one pipeline configuration and each filter and output will receive data from all inputs present.

If you want each file to work as an independent pipeline you need to configure logstash to run [multiple pipelines](https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html) with the `pipelines.yml`.

Per default the `pipelines.yml` will run one pipeline named `main` which will merge all the files inside `/etc/logstash/conf.d`, so you need to change it.

Basically you need something like this:

```auto
- pipeline.id: pipeline-01
  path.config: "/etc/logstash/conf.d/pipeline-01.conf"
- pipeline.id: pipeline-02
  path.config: "/etc/logstash/conf.d/pipeline-02.conf"

```

Are you running lostash as service, right?

---

<div class="post-metadata">

**Author:** ![taniumalloy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taniumalloy/32/136858_2.png) [@taniumalloy](https://discuss.elastic.co/u/taniumalloy)\
**Post date:** [August 17, 2024, 2:39pm UTC](https://discuss.elastic.co/t/two-logstash-http-poller-api-query-configs-are-outputting-documents-to-each-others-indices/365066/3 "2024-08-17T14:39:52Z")

</div>

Thanks so much for the detailed explanation! That really helps.

Yes, I am running logstash as a service in RHEL.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [August 17, 2024, 2:44pm UTC](https://discuss.elastic.co/t/two-logstash-http-poller-api-query-configs-are-outputting-documents-to-each-others-indices/365066/4 "2024-08-17T14:44:25Z")

</div>

You just need to configure the `pipelines.yml` and restar the service then.