# Logstash ingesting from S3, affecting OTHER indices?

**URL:** https://discuss.elastic.co/t/logstash-ingesting-from-s3-affecting-other-indices/300726
**Category:** Logstash
**Created:** [March 25, 2022, 8:14pm UTC](https://discuss.elastic.co/t/logstash-ingesting-from-s3-affecting-other-indices/300726 "2022-03-25T20:14:47Z")
**Posts on this page:** 8
**Page:** 1

<div class="post-metadata">

### Author: ![pritster5](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pritster5/32/103311_2.png) [@pritster5](https://discuss.elastic.co/u/pritster5)
#### Post date: [March 25, 2022, 8:14pm UTC](https://discuss.elastic.co/t/logstash-ingesting-from-s3-affecting-other-indices/300726/1 "2022-03-25T20:14:47Z")

</div>

I'm having a strange issue. I'm using the pipelines feature of Logstash via 2 config files that look like this:

```auto
input {
    s3 {
        bucket => "<BUCKETNAME>"
        region => "us-east-1"
        codec => "json"
        additional_settings => {
            force_path_style => true
            follow_redirects => false
        }
    }
}
output {
    elasticsearch {
        hosts => "http://localhost:9200"
        index => "test.<CLIENTNAME>.output-%{+YYYY.MM}"
        user => logstash_internal
        password => XXXXX
    }
}

```

to ingest data from S3.

However, these 2 .conf files each reference a different bucket. What I don't understand is, upon running Logstash, it seems the indices are sharing data even though each index should correspond to an S3 bucket. As one gets bigger, so does the other.

**How do I keep this from happening and ensure that files from their respective buckets end up in their corresponding index?**

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/2/b2fbad9d77b8a66fe2788ac2773a725ad50cd7dd.png)

---

<div class="post-metadata">

### Author: ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [March 25, 2022, 8:19pm UTC](https://discuss.elastic.co/t/logstash-ingesting-from-s3-affecting-other-indices/300726/2 "2022-03-25T20:19:18Z")

</div>

Can you share your `pipelines.yml` file?

---

<div class="post-metadata">

### Author: ![pritster5](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pritster5/32/103311_2.png) [@pritster5](https://discuss.elastic.co/u/pritster5)
#### Post date: [March 25, 2022, 8:20pm UTC](https://discuss.elastic.co/t/logstash-ingesting-from-s3-affecting-other-indices/300726/3 "2022-03-25T20:20:04Z")

</div>

Sure thing.

It's literally just:

```auto
- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"

```

---

<div class="post-metadata">

### Author: ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [March 25, 2022, 8:24pm UTC](https://discuss.elastic.co/t/logstash-ingesting-from-s3-affecting-other-indices/300726/4 "2022-03-25T20:24:47Z")

</div>

WIth this configuration you have just **one** pipeline, when logstash starts it will merge all the files in the `/etc/logstash/conf.d` path as it was just one file.

Since you are not using conditionals in your output, the data from both inputs will be sent to all the outputs, you need to change the `pipelines.yml` file to use multiple pipelines.

Something like this:

```nohighlight
- pipeline.id: pipeline-one
  path.config: "/etc/logstash/conf.d/pipeline-one.conf

- pipeline.id: pipeline-two
  path.config: "/etc/logstash/conf.d/pipeline-two.conf

```

This will make logstash run both pipelines, but as separated processes, the events of one pipeline won't exist to the other pipeline.

---

<div class="post-metadata">

### Author: ![pritster5](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pritster5/32/103311_2.png) [@pritster5](https://discuss.elastic.co/u/pritster5)
#### Post date: [March 25, 2022, 8:32pm UTC](https://discuss.elastic.co/t/logstash-ingesting-from-s3-affecting-other-indices/300726/5 "2022-03-25T20:32:55Z")

</div>

Ahh, I had no idea that was how it worked. Thanks so much!

---

<div class="post-metadata">

### Author: ![pritster5](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pritster5/32/103311_2.png) [@pritster5](https://discuss.elastic.co/u/pritster5)
#### Post date: [March 25, 2022, 8:33pm UTC](https://discuss.elastic.co/t/logstash-ingesting-from-s3-affecting-other-indices/300726/6 "2022-03-25T20:33:40Z")

</div>

As a follow-up, if I wanted to re-ingest everything I already ingested, would I need to delete the sincedb files and restart logstash?

---

<div class="post-metadata">

### Author: ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [March 25, 2022, 8:36pm UTC](https://discuss.elastic.co/t/logstash-ingesting-from-s3-affecting-other-indices/300726/7 "2022-03-25T20:36:41Z")

</div>

You would need to stop logstash, remove the sincedb files created by the s3 input and start it again, the sincedb file for the s3 input basically just stores the date of the last object that was read from the bucket.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 22, 2022, 8:36pm UTC](https://discuss.elastic.co/t/logstash-ingesting-from-s3-affecting-other-indices/300726/8 "2022-04-22T20:36:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.