# Publish same Log file logs to elastic with different index

**URL:** <https://discuss.elastic.co/t/publish-same-log-file-logs-to-elastic-with-different-index/236511>\
**Category:** Logstash\
**Created:** [June 10, 2020, 12:53pm UTC](https://discuss.elastic.co/t/publish-same-log-file-logs-to-elastic-with-different-index/236511 "2020-06-10T12:53:14Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vinutha\_A\_H](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinutha_a_h/32/48343_2.png) [@Vinutha\_A\_H](https://discuss.elastic.co/u/Vinutha_A_H)\
**Post date:** [June 10, 2020, 12:53pm UTC](https://discuss.elastic.co/t/publish-same-log-file-logs-to-elastic-with-different-index/236511/1 "2020-06-10T12:53:14Z")

</div>

Hi,

We are trying to read a log file twice. Sending one copy as it is (just mapping the key to each field in logs using grok) to kafka/elastic with index xyz and second copy by removing some fields (we are also doing key mapping to each field in logs using grok) to Kafka/elastic with index abcd. The Kafka broker, topic and everything else is same.

I see most of the logs being published to both the index but I also see few missing in either of the index.

So, would like to get a expert view on this. As, if it is the correct thing to do and is logstash expected to work smoothly in this scenario.

---

<div class="post-metadata">

**Author:** ![Vinutha\_A\_H](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinutha_a_h/32/48343_2.png) [@Vinutha\_A\_H](https://discuss.elastic.co/u/Vinutha_A_H)\
**Post date:** [June 11, 2020, 8:52am UTC](https://discuss.elastic.co/t/publish-same-log-file-logs-to-elastic-with-different-index/236511/2 "2020-06-11T08:52:03Z")

</div>

Hi,

Any comments on this topic?

---

<div class="post-metadata">

**Author:** ![Jenni](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jenni/32/29684_2.png) [@Jenni](https://discuss.elastic.co/u/Jenni)\
**Post date:** [June 11, 2020, 11:38am UTC](https://discuss.elastic.co/t/publish-same-log-file-logs-to-elastic-with-different-index/236511/3 "2020-06-11T11:38:56Z")

</div>

I don't see an obvious reason why this wouldn't work, but then again it's a very abstract question. Maybe the Logstash configuration, Elasticsearch field mappings, error logs or examples for the missing events would give us a clue.

---

<div class="post-metadata">

**Author:** ![Vinutha\_A\_H](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinutha_a_h/32/48343_2.png) [@Vinutha\_A\_H](https://discuss.elastic.co/u/Vinutha_A_H)\
**Post date:** [June 11, 2020, 1:13pm UTC](https://discuss.elastic.co/t/publish-same-log-file-logs-to-elastic-with-different-index/236511/4 "2020-06-11T13:13:14Z")

</div>

Thanks Jenni for your response. Logstash config looks as below and dont see any errors in logstash logs.

The index mapped at elastic are 'abcd' and 'abd-efg'  
Please let me know, if in case any issues with these config and mapping.

//  
Input {  
file {  
path =\> ["/local/logs/filename.log"]  
add\_field =\> { "app" =\> "abcd" "DOMAIN" =\> "test"}  
type =\> 'wxyz'  
}  
file {  
path =\> ["/local/logs/filename.log"]  
add\_field =\> { "app\_id" =\> "abc-efg" "DOMAIN" =\> "test"}  
type =\> 'wxy'  
}  
}

filter {  
if [type] == "wxyz" {  
grok {  
match =\> ["message", "%{DATE:date}-%{TIME:time} NAME:%{GREEDYDATA:text} DATA:%{GREEDYDATA:text}"]  
}  
mutate { remove\_field =\> ["message" , "DATA"] }  
mutate { add\_field =\> ["ts" , "%{date} %{time}"] }  
date {  
match =\> ["ts", "yy/MM/dd HH:mm:ss"]  
target =\> "@timestamp"  
}  
}  
if [type] == "wxy" {  
grok {  
match =\> ["message", "%{DATE:date}-%{TIME:time} NAME:%{GREEDYDATA:text} DATA:%{GREEDYDATA:text}"]  
}  
mutate { add\_field =\> ["ts" , "%{date} %{time}"] }  
date {  
match =\> ["ts", "yy/MM/dd HH:mm:ss"]  
target =\> "@timestamp"  
}  
}  
}

output {  
kafka {  
topic\_id =\> ""  
bootstrap\_servers =\> "brokerhost:brokerport"  
codec =\> json {}  
security\_protocol =\> "SSL"  
ssl\_keystore\_location =\> ""  
ssl\_keystore\_password =\> ""  
ssl\_truststore\_location =\> ""  
ssl\_truststore\_password =\> ""  
}  
}

---

<div class="post-metadata">

**Author:** ![Vinutha\_A\_H](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinutha_a_h/32/48343_2.png) [@Vinutha\_A\_H](https://discuss.elastic.co/u/Vinutha_A_H)\
**Post date:** [June 12, 2020, 11:40am UTC](https://discuss.elastic.co/t/publish-same-log-file-logs-to-elastic-with-different-index/236511/5 "2020-06-12T11:40:41Z")

</div>

Hi Jenni, I hope there is nothing wring in the config. Please confirm.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 10, 2020, 11:40am UTC](https://discuss.elastic.co/t/publish-same-log-file-logs-to-elastic-with-different-index/236511/6 "2020-07-10T11:40:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.