# Reindex data is multiplying docs

**URL:** <https://discuss.elastic.co/t/reindex-data-is-multiplying-docs/296920>\
**Category:** Logstash\
**Created:** [February 11, 2022, 12:58am UTC](https://discuss.elastic.co/t/reindex-data-is-multiplying-docs/296920 "2022-02-11T00:58:46Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![cris](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cris/32/101855_2.png) [@cris](https://discuss.elastic.co/u/cris)\
**Post date:** [February 11, 2022, 12:58am UTC](https://discuss.elastic.co/t/reindex-data-is-multiplying-docs/296920/1 "2022-02-11T00:58:46Z")

</div>

Hello community!  
I am working with logstash to reindex some data, I have a problem when I run logstash.service the documents increase a lot in the new index, in the original index I have 3000 documents and I expect like 12000 but in the new index it never stops increasing the number of documents It's like infinity. But when I run only

```auto
/usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/logstash.conf

```

is workin fine This is my logstash setup.

```auto
input {
  elasticsearch {
    hosts => "localhost:9200"
    index => "testingService"
    size => 1
   scroll => "5m"
    docinfo => true
  }
}
filter{
ruby {
           code => '

array2 = []
env1 = event.get("[Spain][Testing][status]")
env2 = event.get("[USA][Testing][status]")
env3 = event.get("[Brazil][Testing][status]")
env4 = event.get("[London][Testing][status]")
array2 << {"status": env1}
array2 << {"status": env2}
array2 << {"status":env3}
array2 << {"status": env4}
      event.to_hash.keys.each{|k|
        if !(k.start_with?("@","timelocal")) then
          event.remove(k)
        end
      }
      event.set("[Login]", array2)

'
    }
split { field => '[Login]' }

}
output {
  elasticsearch {
    hosts => "localhost:9200"
    index => "reindex-data"
  }
 stdout {
   codec => rubydebug
   }

}

```

Thanks for your time

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 11, 2022, 7:33am UTC](https://discuss.elastic.co/t/reindex-data-is-multiplying-docs/296920/2 "2022-02-11T07:33:17Z")

</div>

It may be not infinity, but index the whole documents again as you run the Logstash pipeline once.

Every time` elasticsearch output plugin` send a new document to `reindex-data` index of Elasticsearch, a unique `_id` for the document is generated. There is no automatic tracking function.

To avoid duplication, you have to query source documents only which has not been indexed or use [fingerprint filter plugin](https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html) and set the fingerprint to id of the output document to identify same documents.

---

<div class="post-metadata">

**Author:** ![cris](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cris/32/101855_2.png) [@cris](https://discuss.elastic.co/u/cris)\
**Post date:** [February 15, 2022, 3:01am UTC](https://discuss.elastic.co/t/reindex-data-is-multiplying-docs/296920/3 "2022-02-15T03:01:01Z")

</div>

I was trying with fingerprint filter buy now I only get 1 hit because I have always the same id.

```auto
fingerprint {
        source => ["status"]
        target => "[@metadata][generated_id]"
        concatenate_sources => true
  }
}
output {
  elasticsearch {
    hosts => "localhost:9200"
    index => "prueba-reindex17"
   document_id => "%{[@metadata][fingerprint]}"

```

How can I assigned the id per each split?

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 15, 2022, 3:11am UTC](https://discuss.elastic.co/t/reindex-data-is-multiplying-docs/296920/4 "2022-02-15T03:11:55Z")

</div>

> [@cris](#):
>
> ```auto
> output {
> elasticsearch {
> hosts => "localhost:9200"
> index => "prueba-reindex17"
> document_id => "%{[@metadata][fingerprint]}"
> 
> ```

```auto
output {
  elasticsearch {
    hosts => "localhost:9200"
    index => "prueba-reindex17"
   document_id => "%{[@metadata][generated_id]}"

```

---

<div class="post-metadata">

**Author:** ![cris](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cris/32/101855_2.png) [@cris](https://discuss.elastic.co/u/cris)\
**Post date:** [February 15, 2022, 3:28am UTC](https://discuss.elastic.co/t/reindex-data-is-multiplying-docs/296920/5 "2022-02-15T03:28:26Z")

</div>

Sorry I did not see my error, but I tried now with generated\_id but I get the same , only one hit

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 15, 2022, 3:41am UTC](https://discuss.elastic.co/t/reindex-data-is-multiplying-docs/296920/6 "2022-02-15T03:41:28Z")

</div>

Please share the output using stdout output plugin with rubydebug codec.

---

<div class="post-metadata">

**Author:** ![cris](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cris/32/101855_2.png) [@cris](https://discuss.elastic.co/u/cris)\
**Post date:** [February 15, 2022, 4:41am UTC](https://discuss.elastic.co/t/reindex-data-is-multiplying-docs/296920/7 "2022-02-15T04:41:18Z")

</div>

I only get on rubydebug

```auto
{
     "localTime" => "2022-02-13T13:28:12.947-06:00",
    "@timestamp" => 2022-02-15T04:39:01.548Z,
      "@version" => "1",
         "Login" => {
        "status" => "passed"
    }
}
{
     "localTime" => "2022-02-13T13:28:12.947-06:00",
    "@timestamp" => 2022-02-15T04:39:01.548Z,
      "@version" => "1",
         "Login" => {
        "status" => "passed"
    }
}
{
     "localTime" => "2022-02-13T13:28:12.947-06:00",
    "@timestamp" => 2022-02-15T04:39:01.548Z,
      "@version" => "1",
         "Login" => {
        "status" => "passed"
    }
}
{
     "localTime" => "2022-02-13T13:28:12.947-06:00",
    "@timestamp" => 2022-02-15T04:39:01.548Z,
      "@version" => "1",
         "Login" => {
        "status" => "passed"
    }
}

```

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 15, 2022, 5:10am UTC](https://discuss.elastic.co/t/reindex-data-is-multiplying-docs/296920/8 "2022-02-15T05:10:52Z")

</div>

Because status is identical, fingerprints should be identical. Fingerprint source have to be identical to the events what you want to deduplicate. You need some more fields.

---

<div class="post-metadata">

**Author:** ![cris](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cris/32/101855_2.png) [@cris](https://discuss.elastic.co/u/cris)\
**Post date:** [February 15, 2022, 6:31am UTC](https://discuss.elastic.co/t/reindex-data-is-multiplying-docs/296920/9 "2022-02-15T06:31:37Z")

</div>

Ok, I added other field in the ruby code, like this:

```auto
array2 << {"status": env4, "instance": "Spain"}
array2 << {"status": env4, "instance": "USA"}
array2 << {"status": env4, "instance": "Brazil"}
array2 << {"status": env4, "instance": "London"}

```

and in the fingerprint I change the source for

```auto
fingerprint {
        source => ["instance"]
        target => "[@metadata][generated_id]"
        concatenate_sources => true
  }

```

but I get again other 1 hit ☹  
This is the rubydebug

```auto
{
    "@timestamp" => 2022-02-15T06:28:14.339Z,
     "localTime" => "2022-02-14T21:07:40.329-06:00",
         "Login" => {
           "status" => "passed",
        "instance" => "Spain"
    },
      "@version" => "1"
}
{
    "@timestamp" => 2022-02-15T06:28:14.339Z,
     "localTime" => "2022-02-14T21:07:40.329-06:00",
         "Login" => {
           "status" => "passed",
        "instance" => "USA"
    },
      "@version" => "1"
}
{
    "@timestamp" => 2022-02-15T06:28:14.339Z,
     "localTime" => "2022-02-14T21:07:40.329-06:00",
         "Login" => {
           "status" => "passed",
        "instance" => "Brazil"
    },
      "@version" => "1"
}
{
    "@timestamp" => 2022-02-15T06:28:14.339Z,
     "localTime" => "2022-02-14T21:07:40.329-06:00",
         "Login" => {
           "status" => "passed",
        "instance" => "London"
    },
      "@version" => "1"
}

```

Is correct the source option in fingerprint 🤔 ?

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 15, 2022, 7:09am UTC](https://discuss.elastic.co/t/reindex-data-is-multiplying-docs/296920/10 "2022-02-15T07:09:43Z")

</div>

Turn on metadata and check `[@metadata][generated_id]` is different for different messages.

```auto
output {
  stdout { codec => rubydebug {metadata => true } }
}

```

And if you use "instance" as the source, you'll get only 4 documents for Spain, USA, Brazil, London. Is it your intention? It depends on you what messages are same and what messages are different.

---

<div class="post-metadata">

**Author:** ![cris](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cris/32/101855_2.png) [@cris](https://discuss.elastic.co/u/cris)\
**Post date:** [February 15, 2022, 7:25am UTC](https://discuss.elastic.co/t/reindex-data-is-multiplying-docs/296920/11 "2022-02-15T07:25:24Z")

</div>

oh yes! I get the same generated\_id in all documents

```auto
}
{
     "localTime" => "2022-02-13T07:38:34.758-06:00",
    "@timestamp" => 2022-02-15T07:16:16.887Z,
     "@metadata" => {
              "_index" => "testingService",
                 "_id" => "8ENO834BWwKzUDWhfhep",
               "_type" => "_doc",
        "generated_id" => "7b20fc900c1341ab65ed59a65fcb535c33059315"
    },
      "@version" => "1",
         "Login" => {
        "instancia" => "Spain",
           "status" => "passed"
    }
}
{
     "localTime" => "2022-02-13T07:38:34.758-06:00",
    "@timestamp" => 2022-02-15T07:16:16.887Z,
     "@metadata" => {
              "_index" => "testingService",
                 "_id" => "8ENO834BWwKzUDWhfhep",
               "_type" => "_doc",
        "generated_id" => "7b20fc900c1341ab65ed59a65fcb535c33059315"
    },
      "@version" => "1",
         "Login" => {
        "instancia" => "USA",
           "status" => "passed"
    }
}
{
     "localTime" => "2022-02-13T07:38:34.758-06:00",
    "@timestamp" => 2022-02-15T07:16:16.887Z,
     "@metadata" => {
              "_index" => "testingService",
                 "_id" => "8ENO834BWwKzUDWhfhep",
               "_type" => "_doc",
        "generated_id" => "7b20fc900c1341ab65ed59a65fcb535c33059315"
    },
      "@version" => "1",
         "Login" => {
        "instancia" => "Brazil",
           "status" => "passed"
    }
}
{
     "localTime" => "2022-02-13T07:38:34.758-06:00",
    "@timestamp" => 2022-02-15T07:16:16.887Z,
     "@metadata" => {
              "_index" => "testingService",
                 "_id" => "8ENO834BWwKzUDWhfhep",
               "_type" => "_doc",
        "generated_id" => "7b20fc900c1341ab65ed59a65fcb535c33059315"
    },
      "@version" => "1",
         "Login" => {
        "instancia" => "London",
           "status" => "passed"
    }
}

```

I pretend get 4 different hits per each hit in the original index. One hit for Spain, othe for USA... etc

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 15, 2022, 3:45pm UTC](https://discuss.elastic.co/t/reindex-data-is-multiplying-docs/296920/12 "2022-02-15T15:45:51Z")

</div>

FIRST, you have to decide what information you want to deduplicate documents on.

Then set the information to the source of `fingerprint`.

`source` could be multiple fields.

Something like the following could be the solution.

```auto
fingerprint {
        source => ["localTime","instance"]
        target => "[@metadata][generated_id]"
        concatenate_sources => true
  }

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 15, 2022, 5:14pm UTC](https://discuss.elastic.co/t/reindex-data-is-multiplying-docs/296920/13 "2022-02-15T17:14:20Z")

</div>

Try `source => ["[Login][instancia]" ]`

---

<div class="post-metadata">

**Author:** ![cris](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cris/32/101855_2.png) [@cris](https://discuss.elastic.co/u/cris)\
**Post date:** [February 16, 2022, 12:21am UTC](https://discuss.elastic.co/t/reindex-data-is-multiplying-docs/296920/14 "2022-02-16T00:21:19Z")

</div>

> [@Tomo\_M](#):
>
> `"localTime"`

@Badger @Tomo_M  
Thanks for your help, I solved merging both suggestions

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 16, 2022, 12:21am UTC](https://discuss.elastic.co/t/reindex-data-is-multiplying-docs/296920/15 "2022-03-16T00:21:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.