# Data from indices are shown on other indices **URL:** \ **Category:** Elasticsearch\ **Created:** [March 28, 2021, 9:39pm UTC](https://discuss.elastic.co/t/data-from-indices-are-shown-on-other-indices/268608 "2021-03-28T21:39:00Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![snobysmake](https://avatars.discourse-cdn.com/v4/letter/s/eb8c5e/32.png) [@snobysmake](https://discuss.elastic.co/u/snobysmake)\ **Post date:** [March 28, 2021, 9:39pm UTC](https://discuss.elastic.co/t/data-from-indices-are-shown-on-other-indices/268608/1 "2021-03-28T21:39:00Z")
I seem to be having a strange problem but quite possibly be a misconfiguration on my side. We are trying to monitor AWS S3 logs which have the following structure ```auto s3://bucket/projectA s3://bucket/projectB s3://bucket/projectC s3://bucket/projectN etc.. ``` We have logstash configured for each of these projects in a different configuration file as well as some of the grok expressions are different based on the type of Load Balancer that the project has. For example we have, ```auto /etc/logstash/conf.d/projectA.conf /etc/logstash/conf.d/projectB.conf /etc/logstash/conf.d/projectC.conf /etc/logstash/conf.d/projectN.conf etc.. ``` If I start logstash service it reads all the conf files and populate the indices but sometimes data from ProjectA is seen on ProjectC and so on. I did a fresh start and only started one configuration file at a time and that seems to house the data to it's own indices. Do we need to configure in a different way for a requirement like this? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 29, 2021, 12:03am UTC](https://discuss.elastic.co/t/data-from-indices-are-shown-on-other-indices/268608/2 "2021-03-29T00:03:55Z")
What does a sample Logstash conf look like? Because Logstash will mere all of those files into one big one at run time, unless you use pipelines or something else to segregate things. ---
**Author:** ![snobysmake](https://avatars.discourse-cdn.com/v4/letter/s/eb8c5e/32.png) [@snobysmake](https://discuss.elastic.co/u/snobysmake)\ **Post date:** [March 29, 2021, 1:52am UTC](https://discuss.elastic.co/t/data-from-indices-are-shown-on-other-indices/268608/3 "2021-03-29T01:52:33Z")
Thank you for your response. This is one of the project files under conf.d/ named projectA.conf, ```auto input { s3 { bucket => "load-balancer-logs" prefix => "ProjectA" region => "us-west-2" add_field => { "doctype" => "aws-application-load-balancer-for-projectA" } } } grok { statements } output { elasticsearch { hosts => ["http://localhost:9200"] index => "alb-index-projectA-%{+YYYY.MM.dd}" #user => "user" #password => "password" } } ``` My pipelines.yml has nothing but default entries, ```auto - pipeline.id: main path.config: "/etc/logstash/conf.d/*.conf" ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [March 29, 2021, 2:01am UTC](https://discuss.elastic.co/t/data-from-indices-are-shown-on-other-indices/268608/4 "2021-03-29T02:01:14Z")
So all those conf files Are merged / concatonated i.e they are all merged into a single pipeline, so you need an `if {}` block to make sure you send the right docs to the right index in each of the confs. You should do it for your filter groks too. Something like ``` output { if [prefix] == "ProjectA" { elasticsearch { hosts => ["http://localhost:9200"] index => "alb-index-projectA-%{+YYYY.MM.dd}" #user => "user" #password => "password" } } } ``` The other way to do it us make each its own pipeline by specifically naming each separate in the pipelines.yml as separate pipelines ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 29, 2021, 2:43am UTC](https://discuss.elastic.co/t/data-from-indices-are-shown-on-other-indices/268608/5 "2021-03-29T02:43:00Z")
You can also use sprintf references to make things a lot simpler; ```auto input { s3 { bucket => "load-balancer-logs" prefix => "ProjectA" region => "us-west-2" add_field => { "doctype" => "aws-application-load-balancer-for-%{prefix}" } } } grok { statements } output { elasticsearch { hosts => ["http://localhost:9200"] index => "alb-index-%{prefix}-%{+YYYY.MM.dd}" #user => "user" #password => "password" } } ``` But your groks may be different. ---
**Author:** ![snobysmake](https://avatars.discourse-cdn.com/v4/letter/s/eb8c5e/32.png) [@snobysmake](https://discuss.elastic.co/u/snobysmake)\ **Post date:** [March 29, 2021, 12:17pm UTC](https://discuss.elastic.co/t/data-from-indices-are-shown-on-other-indices/268608/6 "2021-03-29T12:17:34Z")
This is great! I opted for multiple pipelines and it seems to work. ```auto # This file is where you define your pipelines. You can define multiple. # For more information on multiple pipelines, see the documentation: # https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html #- pipeline.id: main # path.config: "/etc/logstash/conf.d/*.conf" - pipeline.id: ProjectA-pipeline path.config: "/etc/logstash/conf.d/ProjectA.conf" - pipeline.id: ProjectB-pipeline path.config: "/etc/logstash/conf.d/ProjectB.conf" - pipeline.id: ProjectN-pipeline path.config: "/etc/logstash/conf.d/ProjectN.conf" ``` You guys are genius! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 26, 2021, 12:17pm UTC](https://discuss.elastic.co/t/data-from-indices-are-shown-on-other-indices/268608/7 "2021-04-26T12:17:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.