# Noob Help with duplicates

**URL:** <https://discuss.elastic.co/t/noob-help-with-duplicates/79076>\
**Category:** Elasticsearch\
**Created:** [March 17, 2017, 6:34pm UTC](https://discuss.elastic.co/t/noob-help-with-duplicates/79076 "2017-03-17T18:34:20Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Crystal\_Bishop](https://avatars.discourse-cdn.com/v4/letter/c/97f17d/32.png) [@Crystal\_Bishop](https://discuss.elastic.co/u/Crystal_Bishop)\
**Post date:** [March 17, 2017, 6:34pm UTC](https://discuss.elastic.co/t/noob-help-with-duplicates/79076/1 "2017-03-17T18:34:20Z")

</div>

Hi Everyone,

I am seeing all messages duplicated when I search in Kibana.

Example:  
Time \_type MessageHeader.MessageId \_id  
March 15th 2017, 12:22:22.344 Rail2\_chatter 2f7176cc-7bcd-4386-bed0-58ce971d90e1 AVrTa88VyOalNeB0\_gFH  
March 15th 2017, 12:22:22.344 Rail2\_chatter 2f7176cc-7bcd-4386-bed0-58ce971d90e1 AVrTa88VyOalNeB0\_gFI  
March 15th 2017, 12:22:22.344 logs 2f7176cc-7bcd-4386-bed0-58ce971d90e1 AVrTa88qyOalNeB0\_gFJ  
March 15th 2017, 12:22:22.344 logs 2f7176cc-7bcd-4386-bed0-58ce971d90e1 AVrTa88qyOalNeB0\_gFK

I have a unique field [MessageHeader][MessageId] where I only want to see that message once. These four message are the same (even the timestamps) with the exception of the \_id and \_type fields.

Here is the logstash config, redacted to protect the innocent servers:  
input {  
rabbitmq {  
arguments =\> { 'x-ha-policy' =\> all }  
... (stuff removed)  
queue =\> 'logstash'  
vhost =\> 'delphi'  
}  
rabbitmq {  
arguments =\> { 'x-ha-policy' =\> all }  
... (stuff removed)  
queue =\> 'poshost01'  
vhost =\> 'delphi'  
}  
}  
filter {  
if [MessageHeader] {  
mutate {  
replace =\> { "[MessageHeader][ProcessedTime]" =\> "%{@timestamp}" }  
}  
}  
}  
output {  
if [SourceHeader][SourceType] == "Rail2" {  
if [message][type] == "HEARTBEAT" {  
elasticsearch {  
hosts =\> "http://[elasticsearch]:[port]"  
index =\> "[indexname]"  
document\_type =\> "Rail2\_heartbeat"  
}  
}  
if [message][type] == "CHATTER" {  
elasticsearch {  
hosts =\> "[http://es01-lab:9200](http://es01-lab:9200)"  
index =\> "[indexname]"  
document\_type =\> "Rail2\_chatter"  
}  
}  
}  
if [SourceHeader][SourceType] == "POS.Host" {  
if [message][type] == "HEARTBEAT" {  
elasticsearch {  
hosts =\> "http://[elasticsearch]:[port]"  
index =\> "[indexname]"  
document\_type =\> "POS.Host\_heartbeat"  
}  
}  
if [message][type] == "CHATTER" {  
elasticsearch {  
hosts =\> "http://[elasticsearch]:[port]"  
index =\> "[indexname]"  
document\_type =\> "POS.Host\_chatter"  
}  
}  
}

stdout { codec =\> rubydebug }  
}

Looking at the config, I'm not sure where the doc type 'logs' came from, I would think we would only see the message once in the doc type 'Rail2\_chatter'. Could the duplicates be because of the double input from rabbitmq?

How can I change the config to filter for unique instances of [MessageHeader][MessageId] ?

Thanks muchly!

---

<div class="post-metadata">

**Author:** ![eperry](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eperry/32/551_2.png) [@eperry](https://discuss.elastic.co/u/eperry)\
**Post date:** [March 18, 2017, 1:14pm UTC](https://discuss.elastic.co/t/noob-help-with-duplicates/79076/2 "2017-03-18T13:14:14Z")

</div>

not sure about the [message][type} if statements, as the message field is using a string, and I don't see you converting it with the [json] filter. Though I am not familiar with your data format.

I think there might be ways of simplifying your output section to be easier to read  
For examole, on every Inputut I add a field called "dst\_index" then in my elasticsearch output

I can do

> if [dst\_indes] =~ /heartbeat/

> elkasticsearch{}

> else

> elasticsearch{  
> index =\> "%{dst\_index}"

> }

This allows me to treat the one heartbeat index differently

Also I like to add a tag to my input / filters and heck even outputs when debugging so I can see what block is processing the data.

hope this gives you some idea's

oh and if data is duplicated it is probably hitting multiple output statements

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 15, 2017, 1:14pm UTC](https://discuss.elastic.co/t/noob-help-with-duplicates/79076/3 "2017-04-15T13:14:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.