# Duplicate documents in Elasticsearch?

**URL:** <https://discuss.elastic.co/t/duplicate-documents-in-elasticsearch/99165>\
**Category:** Elasticsearch\
**Created:** [September 1, 2017, 7:50pm UTC](https://discuss.elastic.co/t/duplicate-documents-in-elasticsearch/99165 "2017-09-01T19:50:47Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![irom77](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/irom77/32/12976_2.png) [@irom77](https://discuss.elastic.co/u/irom77)\
**Post date:** [September 1, 2017, 7:50pm UTC](https://discuss.elastic.co/t/duplicate-documents-in-elasticsearch/99165/1 "2017-09-01T19:50:47Z")

</div>

I am sending one document to Elastic 5.5 from Logstash. But I am getting duplicate documents, two documents with different IDs but same fields. Below I am showing only index , type and id.

```
"hits": {
    "total": 2,
    "max_score": 1,
    "hits": [
      {
        "_index": "logstash-2017.09.01",
        "_type": "threat",
        "_id": "AV4-7XhNlO-DC0yfkKOf",
        "_score": 1
      },
      {
        "_index": "logstash-2017.09.01",
        "_type": "threat",
        "_id": "AV4-7XhklO-DC0yfkKOg",
        "_score": 1
      }
    ]
  }
```

---

<div class="post-metadata">

**Author:** ![Wayne\_Taylor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wayne_taylor/32/45984_2.png) [@Wayne\_Taylor](https://discuss.elastic.co/u/Wayne_Taylor)\
**Post date:** [September 1, 2017, 8:15pm UTC](https://discuss.elastic.co/t/duplicate-documents-in-elasticsearch/99165/2 "2017-09-01T20:15:35Z")

</div>

Can you share your configuration?

---

<div class="post-metadata">

**Author:** ![irom77](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/irom77/32/12976_2.png) [@irom77](https://discuss.elastic.co/u/irom77)\
**Post date:** [September 1, 2017, 8:24pm UTC](https://discuss.elastic.co/t/duplicate-documents-in-elasticsearch/99165/3 "2017-09-01T20:24:47Z")

</div>

Sure, this is only one node, see below.

**{  
"logstash-2017.09.01": {  
"settings": {  
"index": {  
"creation\_date": "1504278620020",  
"number\_of\_shards": "5",  
"number\_of\_replicas": "1",  
"uuid": "Bz6r4KcBRDWxwgsTDU5Epg",  
"version": {  
"created": "5050299"  
},  
"provided\_name": "logstash-2017.09.01"  
}  
}  
}  
}**

In fact I am using popular [github.com/deviantony/docker-elk](http://github.com/deviantony/docker-elk), vanilla config (easy to recreate dokcer containers)

```
version: '2'

services:

  elasticsearch:
    build: elasticsearch/
    volumes:
      - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
      - elk

  logstash:
    build: logstash/
    volumes:
      - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml
      - ./logstash/pipeline:/usr/share/logstash/pipeline
    ports:
      - "11514:11514/udp"
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
      - elk
    depends_on:
      - elasticsearch

  kibana:
    build: kibana/
    volumes:
      - ./kibana/config/:/usr/share/kibana/config
    ports:
      - "5601:5601"
    networks:
      - elk
    depends_on:
      - elasticsearch

networks:

  elk:
    driver: bridge
```

---

<div class="post-metadata">

**Author:** ![irom77](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/irom77/32/12976_2.png) [@irom77](https://discuss.elastic.co/u/irom77)\
**Post date:** [September 1, 2017, 8:27pm UTC](https://discuss.elastic.co/t/duplicate-documents-in-elasticsearch/99165/4 "2017-09-01T20:27:55Z")

</div>

Just to prove the documents are identical:

```
$ curl -XGET '10.254.253.100:9200/logstash-2017.09.01/_search?q=_id:"AV4-7XhklO-DC0yfkKOg"&pretty' > es1
$ curl -XGET '10.254.253.100:9200/logstash-2017.09.01/_search?q=_id:"AV4-7XhNlO-DC0yfkKOf"&pretty' > es2
$ diff es1 es2
16c16
< "_id" : "AV4-7XhklO-DC0yfkKOg",
---
> "_id" : "AV4-7XhNlO-DC0yfkKOf",

```

and this is not a logstash problem because I am loggoing to stdout and getting only one there

```
output {
        if [type] == "threat" {
               
                elasticsearch { 
                        hosts => ["10.254.253.100:9200"] 
                        manage_template => false

                }
                stdout { codec => rubydebug }
        }
        
        
}

$ docker logs dockerelk_logstash_1 | grep 8.8.8.8

               "SourceIP" => "8.8.8.8",
```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 2, 2017, 9:02pm UTC](https://discuss.elastic.co/t/duplicate-documents-in-elasticsearch/99165/5 "2017-09-02T21:02:21Z")

</div>

What does the entire Logstash config look like?

---

<div class="post-metadata">

**Author:** ![irom77](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/irom77/32/12976_2.png) [@irom77](https://discuss.elastic.co/u/irom77)\
**Post date:** [September 3, 2017, 4:16pm UTC](https://discuss.elastic.co/t/duplicate-documents-in-elasticsearch/99165/6 "2017-09-03T16:16:37Z")

</div>

I added uuif fingerprinting to logstash and now the problem disappeared

```
filter {
        if [type] == "threat" {

                
               fingerprint {
                   target => "%{[@metadata][uuid]}"
                   method => "UUID"
               }
               
       }
       
}    
output {
            if [type] == "threat" {
                   
                    elasticsearch { 
                            hosts => ["10.254.253.100:9200"] 
                            manage_template => false
                            document_id => "%{[@metadata][uuid]}"
                            index => "debug"
                            template_name => "debug"

                    }
                    stdout { codec => rubydebug }
            }     
    }

```

However it is strange because id is like old one i.e. "\_id": "AV5DJ6A8Ap8QZuUUMBEC" but meta-data shows different. I expetced uudi to replace id

```
"%{": {
            "@metadata": {
              "uuid": {
                "}": "46a45cc5-365d-40c5-af89-105d58bc6160"
              }
            }
          }
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 1, 2017, 4:17pm UTC](https://discuss.elastic.co/t/duplicate-documents-in-elasticsearch/99165/7 "2017-10-01T16:17:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.