# Logs duplication

**URL:** <https://discuss.elastic.co/t/logs-duplication/247795>\
**Category:** Logstash\
**Created:** [September 7, 2020, 5:37pm UTC](https://discuss.elastic.co/t/logs-duplication/247795 "2020-09-07T17:37:46Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![KOTOXJle6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kotoxjle6/32/131136_2.png) [@KOTOXJle6](https://discuss.elastic.co/u/KOTOXJle6)\
**Post date:** [September 7, 2020, 5:37pm UTC](https://discuss.elastic.co/t/logs-duplication/247795/1 "2020-09-07T17:37:46Z")

</div>

Hello everyone!

Suddenly, the elastic began to double logs. Each log falls into its own index and the duplicate falls into the unkown\_messages index. It seems that this started after adding new indexes, but rolling back the config did not help.

LS configs are stored in /etc/logstash/conf. d/, there are 3 files:  
00-input. conf  
10-filter.conf  
20-output.conf

20-output is like this:

> output {
> 
> # Configuring Windows Domain Controllers log output
> 
> if "windc" in [tags] {  
> elasticsearch {  
> hosts =\> ["10.199.5.104:9200","10.199.5.105:9200","10.199.5.106:9200"]  
> index =\> "windc-%{+YYYY.MM.dd}"  
> }  
> }
> 
> # Configuring ksmg log output
> 
> if "ksmg" in [tags] {  
> elasticsearch {  
> hosts =\> ["10.199.5.104:9200","10.199.5.105:9200","10.199.5.106:9200"]  
> index =\> "ksmg-%{+YYYY.MM.dd}"  
> }  
> }
> 
> # Configuring the output of Exchange mail logs
> 
> if "exchange-mtlog" in [tags] {  
> elasticsearch {  
> hosts =\> ["10.199.5.104:9200","10.199.5.105:9200","10.199.5.106:9200"]  
> index =\> "exchange-mtlog-%{+YYYY.MM.dd}"
> 
> }  
> }
> 
> # Configuring the output of other logs
> 
> else {  
> elasticsearch {  
> hosts =\> ["10.199.5.104:9200","10.199.5.105:9200","10.199.5.106:9200"]  
> index =\> "unknown\_messages"  
> }  
> }
> 
> }

After running logstash, the service log contains these messages::

> [WARN][logstash.outputs.elasticsearch][main][e9b556ef36eb5a3aa1e07673fcd36a804aeecfb9a6de701dcf08a41e172da77a] Could not index event to Elasticsearch. {:status=\>400,:action=\>["index", {:\_id=\>nil, :\_index=\>"unknown\_messages", :routing=\>nil, :\_type=\>"\_doc"},#LogStash::Event:0x682b1390], :response=\>{"index"=\>{"\_index"=\>"unknown\_messages","\_type"=\>"\_doc", "\_id"=\>"GcUcXnQBuxL4OHJpyi4Y", "status"=\>400, "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse field [source] of type [text] in document with id 'GcUcXnQBuxL4OHJpyi4Y'". Preview of field's value: '{domain=DPC-RDS}'", "caused\_by"=\>{"type"=\> "illegal\_state\_exception", "reason"=\> " Can't get text on a START\_OBJECT

Preview of field's value: '{domain=DPC-RDS}'" - this part changes depending on the incoming log, the rest is identical. Ah and ID respectively, too, different. And logs with different IDS end up in indexes. The tags for which logs should fall into the corresponding indexes are found in both the correct indexes and the unknown\_messages index.

Has anyone ever encountered this? Which way to dig?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 7, 2020, 9:11pm UTC](https://discuss.elastic.co/t/logs-duplication/247795/2 "2020-09-07T21:11:06Z")

</div>

What you have is

```
if "windc" in [tags] {
}
if "ksmg" in [tags] {
}
if "exchange-mtlog" in [tags] {
} else {
}

```

and it sounds like what you want is

```
if "windc" in [tags] {
} else if "ksmg" in [tags] {
} else if "exchange-mtlog" in [tags] {
} else {
}
```

---

<div class="post-metadata">

**Author:** ![KOTOXJle6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kotoxjle6/32/131136_2.png) [@KOTOXJle6](https://discuss.elastic.co/u/KOTOXJle6)\
**Post date:** [September 8, 2020, 5:12am UTC](https://discuss.elastic.co/t/logs-duplication/247795/3 "2020-09-08T05:12:42Z")

</div>

Yes, you right. I changed it to be same as in filter. conf for no reason...Thank you very much!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 6, 2020, 5:12am UTC](https://discuss.elastic.co/t/logs-duplication/247795/4 "2020-10-06T05:12:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.