# Data going into wrong index

**URL:** <https://discuss.elastic.co/t/data-going-into-wrong-index/124653>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 19, 2018, 10:13pm UTC](https://discuss.elastic.co/t/data-going-into-wrong-index/124653 "2018-03-19T22:13:54Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![chandukreddi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chandukreddi/32/29241_2.png) [@chandukreddi](https://discuss.elastic.co/u/chandukreddi)\
**Post date:** [March 19, 2018, 10:13pm UTC](https://discuss.elastic.co/t/data-going-into-wrong-index/124653/1 "2018-03-19T22:13:54Z")

</div>

Hello Experts,

Different type of logs going into single index, technically it should go to different new (defined) indexes.

I am differentiating 2 different log files with TAG and then applying filter on TAG in logstash.

_ **my filebeat.yaml file config:** _

- type: log  
enabled: true  
paths:

- type: log  
paths:

ouput to logstash

In my Logstash I have 2 config file one for cassandralog and another one for swiftproxylog

_ **Cassandra.conf** _  
input {  
beats {  
port =\> 5044  
}  
}

filter {  
if "cassandra\_test" in [tags]{  
grok {  
patterns\_dir =\> ["path"]  
break\_on\_match =\> true  
match =\> {  
"message" =\> [  
#"%{CASS\_COMPACTION\_LARGE\_KEY}",  
"%{CASS\_SLAB\_POOL\_CLEANER\_1}",  
"%{CASS\_SLAB\_POOL\_CLEANER\_2}",  
# "%{CASS\_MEMTABLE\_FLUSH\_START}",  
# "%{CASS\_MEMTABLE\_FLUSH\_COMPLETE}",  
"%{CASS\_BATCH\_STATEMENT}",  
"%{CASS\_SIMS\_TOMBSTONE}",  
"%{CASS\_COMPACTION\_COMPLETE}",  
"%{CASS\_GC\_GRACE}",  
"%{CASS\_SERVICE\_THREAD\_PENDING}"  
]  
}  
add\_tag =\> ["cass\_parsed"]  
}

output {  
elasticsearch {  
hosts =\> "host\_ip:9200"  
index =\> "prd-log-%{+YYYY.MM.dd.HH}-000001"  
template =\> "cass\_log\_sizing\_2.json"  
template\_name =\> "cassandra\_log"  
template\_overwrite =\> true  
}  
}

_ **Swiftproxylog.conf** _

input {  
beats {  
port =\> 5044  
}  
}

filter {  
if "swift\_proxy\_test" in [tags]{  
grok {  
patterns\_dir =\> ["path"]  
break\_on\_match =\> true  
match =\> { "message" =\> ["%{SWIFT\_P\_ALL}"] }  
add\_tag =\> ["swift\_all\_parsed"]  
}

if "swift\_all\_parsed" not in [tags] {  
grok {  
patterns\_dir =\> ["path"]  
match =\> { "message" =\> ["%{SWIFT\_P\_204\_499}"] }  
add\_tag =\> ["swift\_rest"]  
}  
}

output {  
elasticsearch {  
hosts =\> "host\_ip:9200"  
index =\> "swift-proxy-log-%{+YYYY.MM.dd.HH}"  
manage\_template =\> "false"  
}  
}

when ever I ran logstash on _ **Cassandra.conf** _ all 2 files data going into prd-log-\* index and if I ran  
_ **Swiftproxylog.conf** _ all the data going into swift-proxy-log-\* index it supposed to go it's own defined index as per filter TAG and elasticsearch output config.

But I do notice one thing, eventhough it's going to same index but I see tags are correct the only problem I see is data mix into one index.

below is the mixing data image:

This index shouldn't have this data

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/9/097473ee9e7b04938e951d0f3c58c23245cd0955.png)

This index should have only this data not above one

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/6/36c1485a69358ccff385499eb68259982e4fc82e.png)

Please advice/correct me if I am doing anything wrong here.  
Thanks  
Chandra

---

<div class="post-metadata">

**Author:** ![chandukreddi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chandukreddi/32/29241_2.png) [@chandukreddi](https://discuss.elastic.co/u/chandukreddi)\
**Post date:** [March 20, 2018, 9:39pm UTC](https://discuss.elastic.co/t/data-going-into-wrong-index/124653/2 "2018-03-20T21:39:16Z")

</div>

@ .. Can anyone help me on this?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 20, 2018, 10:00pm UTC](https://discuss.elastic.co/t/data-going-into-wrong-index/124653/3 "2018-03-20T22:00:32Z")

</div>

I answered the version of this you put in the logstash thread.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 20, 2018, 10:13pm UTC](https://discuss.elastic.co/t/data-going-into-wrong-index/124653/4 "2018-03-20T22:13:36Z")

</div>

Please keep your questions to one thread, having multiple makes it harder for us to assist you 🙂 [Data going into wrong index-why](https://discuss.elastic.co/t/data-going-into-wrong-index-why/124869/2)

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 20, 2018, 10:13pm UTC](https://discuss.elastic.co/t/data-going-into-wrong-index/124653/5 "2018-03-20T22:13:40Z")

</div>