# Bulk Upsert isn't Importing All Records

**URL:** <https://discuss.elastic.co/t/bulk-upsert-isnt-importing-all-records/287011>\
**Category:** Elasticsearch\
**Created:** [October 18, 2021, 3:05pm UTC](https://discuss.elastic.co/t/bulk-upsert-isnt-importing-all-records/287011 "2021-10-18T15:05:07Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![selectedpixel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/selectedpixel/32/92792_2.png) [@selectedpixel](https://discuss.elastic.co/u/selectedpixel)\
**Post date:** [October 18, 2021, 3:05pm UTC](https://discuss.elastic.co/t/bulk-upsert-isnt-importing-all-records/287011/1 "2021-10-18T15:05:07Z")

</div>

I'm doing a bulk upsert using the Elasticsearch NodeJS library Helper classes.

```auto
const response = await ELSClient.helpers.bulk({
      concurrency: 20,
      datasource: data,
      onDocument (doc) {
        return [{ update: { _index: index, _id: doc.id } },
          { doc_as_upsert: true }
        ]
      }
    })
    if (response.failed > 0) {
      console.log('FAILED', response.failed)
    }

```

I'm upserting about 4.4 million records, 5000 per batch, but I'm only able to get 3.8 million records to be created. When I log the response I don't see any error messages stating that any documents failed.

When I initially ran the upsert only about 3 million records were created. The second time I ran it an additional 800k were created. The server is running at a pretty high CPU percentage, but isn't spiking over 100%, non of the CPU credits are being used.

The documents themselves are pretty small. Maybe 10 fields. These records are coming from a Postgres JOIN table so I know the IDs are unique.

I understand this is a very broad issue with lots of possible issue. I'm just looking for some suggestions to try and figure out how to debug this or what could be the possible issue.

---

<div class="post-metadata">

**Author:** ![joao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joao/32/83447_2.png) [@joao](https://discuss.elastic.co/u/joao)\
**Post date:** [October 18, 2021, 8:17pm UTC](https://discuss.elastic.co/t/bulk-upsert-isnt-importing-all-records/287011/2 "2021-10-18T20:17:40Z")

</div>

Hi,

Have you tried with a value less than 5k? For example, half.

---

<div class="post-metadata">

**Author:** ![selectedpixel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/selectedpixel/32/92792_2.png) [@selectedpixel](https://discuss.elastic.co/u/selectedpixel)\
**Post date:** [October 18, 2021, 8:31pm UTC](https://discuss.elastic.co/t/bulk-upsert-isnt-importing-all-records/287011/3 "2021-10-18T20:31:45Z")

</div>

We’ll I initially tried it with 10k. So 5k is already half.

---

<div class="post-metadata">

**Author:** ![selectedpixel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/selectedpixel/32/92792_2.png) [@selectedpixel](https://discuss.elastic.co/u/selectedpixel)\
**Post date:** [October 29, 2021, 11:44am UTC](https://discuss.elastic.co/t/bulk-upsert-isnt-importing-all-records/287011/4 "2021-10-29T11:44:17Z")

</div>

In the end this was an issue with my Postgres Query and had nothing to do with Elasticsearch. I was not ordering my query uniquely enough and thus it was not able to get through all the items.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 26, 2021, 11:44am UTC](https://discuss.elastic.co/t/bulk-upsert-isnt-importing-all-records/287011/5 "2021-11-26T11:44:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.