# ES-to-ES reindexing: why documents are deleted in destination index?

**URL:** <https://discuss.elastic.co/t/es-to-es-reindexing-why-documents-are-deleted-in-destination-index/83389>\
**Category:** Logstash\
**Created:** [April 24, 2017, 9:23am UTC](https://discuss.elastic.co/t/es-to-es-reindexing-why-documents-are-deleted-in-destination-index/83389 "2017-04-24T09:23:33Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![KIVagant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kivagant/32/7182_2.png) [@KIVagant](https://discuss.elastic.co/u/KIVagant)\
**Post date:** [April 24, 2017, 9:23am UTC](https://discuss.elastic.co/t/es-to-es-reindexing-why-documents-are-deleted-in-destination-index/83389/1 "2017-04-24T09:23:33Z")

</div>

Hello.

I have the next configuration file for the AWS ES reindexing:

```
input {
  elasticsearch {
    hosts => "search-.....es.amazonaws.com:443"
    index => "logstash-source-index*"
    #query => "*"
    size => 500
    scroll => "5m"
    docinfo => true
    ssl => true
  }
}

filter {
}

output {
    if [type] == "tablet-heartbeat-prepared" {
        stdout {
            codec => rubydebug
        }
        elasticsearch {
           document_id => "%{id}"
           hosts => "search-......es.amazonaws.com:443"
           manage_template => false
           ssl => true
           flush_size => 250000
           index => "logstash-destination-index"
           document_type => "%{type}"
        }
    }
}

```

I am using next command to run this config:

```
sudo time /usr/share/logstash/bin/logstash --path.settings="/etc/logstash" --log.level=debug --config.debug --path.logs=/etc/logstash -f /etc/logstash/conf.d/es-to-es.conf

```

After execution I don't see any errors inside `/etc/logstash/logstash-plain.log`. And I see good documents in stdout. But in the destination index I have many deleted documents and just small pie of normal data.

```
https://search-......es.amazonaws.com/_cat/indices/?v

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open logstash-destination-index L3WX...PQ 5 1 64 388 1.2mb 1.2mb

```

Why documents are deleted?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 24, 2017, 9:45am UTC](https://discuss.elastic.co/t/es-to-es-reindexing-why-documents-are-deleted-in-destination-index/83389/2 "2017-04-24T09:45:59Z")

</div>

Perhaps because ES doesn't actually update documents but rather deletes old ones and creates new ones? Hence, updating a document will increase the "deleted" counter until the segment the now obsolete document lived in is merged and the document is _actually_ deleted from the store.

---

<div class="post-metadata">

**Author:** ![KIVagant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kivagant/32/7182_2.png) [@KIVagant](https://discuss.elastic.co/u/KIVagant)\
**Post date:** [April 24, 2017, 11:22am UTC](https://discuss.elastic.co/t/es-to-es-reindexing-why-documents-are-deleted-in-destination-index/83389/3 "2017-04-24T11:22:42Z")

</div>

So, if I understand correctly, the problem with possible duplicates. But I see a normal "id" field in stdout for each document. And it's a new index – how these documents can be updated?

The typical document has unique ID in the `logstash-source-index`:

```
{
...
                   "@version" => "1",
                         "id" => "6d749309-75e2-4848-a197-a91c57ba2d43",
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 22, 2017, 11:24am UTC](https://discuss.elastic.co/t/es-to-es-reindexing-why-documents-are-deleted-in-destination-index/83389/4 "2017-05-22T11:24:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.