# Elasticsearch BULK API with .NET for upsert

**URL:** <https://discuss.elastic.co/t/elasticsearch-bulk-api-with-net-for-upsert/370003>\
**Category:** Elasticsearch\
**Created:** [November 4, 2024, 12:11pm UTC](https://discuss.elastic.co/t/elasticsearch-bulk-api-with-net-for-upsert/370003 "2024-11-04T12:11:33Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![Moni\_Hazarika](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moni_hazarika/32/124158_2.png) [@Moni\_Hazarika](https://discuss.elastic.co/u/Moni_Hazarika)\
**Post date:** [November 4, 2024, 12:11pm UTC](https://discuss.elastic.co/t/elasticsearch-bulk-api-with-net-for-upsert/370003/1 "2024-11-04T12:11:33Z")

</div>

Hi Team,

I am using .NET 8 and ES Version="8.15.6". I an trying to use bulk api with a logic for upsert based on the document id. I tried the below index, update commands from Kibana as per documentation and they work fine

```auto
POST _bulk
{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1" }
{ "update" : {"_id" : "1", "_index" : "test"} }
{ "doc" : {"field2" : "value2"} }

```

However when I try to do the same from my C# .NET code I am seeing a weird behavior. I see 2 docs indexed, 1 like below

```auto
{
        "_index": "MyIndex",
        "_id": "42r-9pIBQ0r3zUFWk2vX",
        "_score": 1,
        "_source": {
          "update": {
            "_index": "MyIndex",
            "_id": "1234"
          }
        }
      }

```

and other document with actual data inside the source. So if I have like 10 docs to be indexed what I see in elastic is 20 docs getting indexed

```auto
{
        "_index": "MyIndex",
        "_id": "5Gr-9pIBQ0r3zUFWk2vX",
        "_score": 1,
        "_source": {
          "doc": {
                "id": "1234",
                //followed by rest of the fields
         }}
}

```

Below is the code

```auto
List<object> bulkIndexOperations
// Check if the document exists in Elasticsearch
      var existsResponse = await _elasticsearchClient.GetAsync<object>(elasticsearchId, idx => idx.Index(indexName));

      if (existsResponse.Found)
      {
        // Prepare an update operation - index operation followed by source
        bulkIndexOperations.Add(new { Update = new { _index = indexName, _id = documentId } });
        bulkIndexOperations.Add(new { doc = document });
      }
      else
      {
        // Prepare an index operation
        bulkIndexOperations.Add(new { Index = new { _index = indexName, _id = documentId } });
        bulkIndexOperations.Add(document);
      }
var bulkResponse = await _elasticsearchClient.BulkAsync(b => b
                    .Index(indexName)
                    .IndexMany(bulkIndexOperations));

```

Any idea why that weird extra document is getting indexed which has no source?

Thanks,  
Moni

---

<div class="post-metadata">

**Author:** ![Moni\_Hazarika](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moni_hazarika/32/124158_2.png) [@Moni\_Hazarika](https://discuss.elastic.co/u/Moni_Hazarika)\
**Post date:** [November 5, 2024, 3:14am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-api-with-net-for-upsert/370003/2 "2024-11-05T03:14:43Z")

</div>

when I run the POST \_bulk directly from Kibana and do a search I see just the documents I inserted.  
Does anyone has any reference around how to use for bulk apis in .NET with capability of differentiating insert and update ?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [November 5, 2024, 6:17am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-api-with-net-for-upsert/370003/3 "2024-11-05T06:17:26Z")

</div>

I'm not a c# dev, but it looks strange to me that you are getting the document by elasticsearchId and then indexing using documentId.

But I'm wondering why you want to do a difference between the first insert (when the document doesn't exist) and the second operation. As you are sending in both cases the same full document AFAICS.

---

<div class="post-metadata">

**Author:** ![Moni\_Hazarika](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moni_hazarika/32/124158_2.png) [@Moni\_Hazarika](https://discuss.elastic.co/u/Moni_Hazarika)\
**Post date:** [November 5, 2024, 6:25am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-api-with-net-for-upsert/370003/4 "2024-11-05T06:25:13Z")

</div>

Thanks @dadoonet for the reply. This is what I am trying to do and I am not able to figure out how to achieve it using ES bulk API.  
I am doing below when using the single API but since I have 300K+ documents I want to use the \_bulk API with appropriate batch size. Basically search for the documentId, if present update it else insert. Right now its not a partial update I understand with the updated fields alone since that is something can't be found out easily. But my requirement is if a document by the ID already exists I should not end up inserting it again but update it.

```auto
string documentId = (string)documentType.GetProperty("Id")?.GetValue(document);
                        if (!string.IsNullOrEmpty(documentId))
                        {
                            var elasticsearchId = new Id(documentId);
                            var existsResponse = await _elasticsearchClient.GetAsync<object>(
                                        elasticsearchId,
                                        idx => idx.Index(indexName));

                                    // Upsert logic based on Id
                                    if (existsResponse.Found) // If the document exists, update it
                                    {
                                        var updateResponse = await UpdateDocumentAsync(elasticsearchId, indexName, document);
                                        if (updateResponse.IsValidResponse)
                                        {
                                            documentIndexed = true; // Successfully updated
                                            indexedCount++; // Increment count for successful updates
                                        }
                                        else
                                        {
                                            _logger.LogError("Failed to update document from {JsonFile}: Error: {DebugInformation}. Retrying...", jsonFile, updateResponse.DebugInformation);
                                        }
                                    }
                                    else // If the document does not exist, insert it
                                    {
                                        var indexResponse = await IndexDocumentAsync(document, indexName, documentId);
                                        if (indexResponse.IsValidResponse)
                                        {
                                            documentIndexed = true; // Successfully indexed
                                            indexedCount++; // Increment count for successful indexing
                                        }
                                        else
                                        {
                                            _logger.LogError("Failed to index document from {JsonFile}: Error: {DebugInformation}. Retrying...", jsonFile, indexResponse.DebugInformation);
                                        }
                                    }

```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [November 5, 2024, 6:56am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-api-with-net-for-upsert/370003/5 "2024-11-05T06:56:39Z")

</div>

Why do you want to update instead of inserting again? Behind the scene you will end up reindexing again the document. So in 99% of the use cases, calling update is useless.

---

<div class="post-metadata">

**Author:** ![Moni\_Hazarika](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moni_hazarika/32/124158_2.png) [@Moni\_Hazarika](https://discuss.elastic.co/u/Moni_Hazarika)\
**Post date:** [November 5, 2024, 7:24am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-api-with-net-for-upsert/370003/6 "2024-11-05T07:24:40Z")

</div>

ok lets say we forget about the upsert requirement here and I do a bulk API call for insert only, which is like the else below

```auto
else
      {
        // Prepare an index operation
        bulkIndexOperations.Add(new { Index = new { _index = indexName, _id = elasticsearchId } });
        bulkIndexOperations.Add(document);
      }

```

But like I mentioned my issue is why am I see 2 docs in elastic vs 1 for the same id?

---

<div class="post-metadata">

**Author:** ![Moni\_Hazarika](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moni_hazarika/32/124158_2.png) [@Moni\_Hazarika](https://discuss.elastic.co/u/Moni_Hazarika)\
**Post date:** [November 5, 2024, 8:00am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-api-with-net-for-upsert/370003/7 "2024-11-05T08:00:46Z")

</div>

Thanks @dadoonet  
So this is the issue. Now I removed the if-else and just doing an Index(no update), by deleting the index from Elastic(so no prev data is available)

```auto
string documentId = ((ModelRecord)document).Id; //ModelRecord is the POCO which has Id field and the unique identifier 
var elasticsearchId = new Id(documentId);
bulkIndexOperations.Add(new { Index = new { _index = indexName, _id = elasticsearchId } });
bulkIndexOperations.Add(document);
if (bulkIndexOperations.Count >= batchSize)
    {
      var indexedCount = await ExecuteBulkIndexingAsync(bulkIndexOperations, indexName);
      bulkIndexOperations.Clear(); // Clear the list for the next batch
    }

```

Inside ExecuteBulkIndexingAsync() I call the bulk API

```auto
var bulkResponse = await _elasticsearchClient.BulkAsync(b => b
                    .Index(indexName)
                    .IndexMany(bulkIndexOperations));

```

I still see double the doc count since I have something like below in Elastic

```auto
"hits": {
    "total": {
      "value": 168630,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "MyIndex",
        "_id": "TJ5N-5IBk9raR26I2YG_",
        "_score": 1,
        "_source": {
          "index": {
            "_index": "MyIndex",
            "_id": "2018-17136"
          }
        }
      },
      {
        "_index": "MyIndex",
        "_id": "2018-17136",
        "_score": 1,
        "_source": { //the actual fields from the ModelRecord POCO
          "id": "2018-17136",
          "fieldA": true,
          "fieldB": false
        }
      }

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 5, 2024, 8:07am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-api-with-net-for-upsert/370003/8 "2024-11-05T08:07:37Z")

</div>

> [@Moni\_Hazarika](#):
>
> ```auto
> "_id": "2018-17136",
> "_score": 1,
> "_source": { //the actual fields from the ModelRecord POCO
> "id": "2018-17136",
> 
> ```

It looks like the `id` field is used for the document id.

> [@Moni\_Hazarika](#):
>
> ```auto
> "_id": "TJ5N-5IBk9raR26I2YG_",
> "_score": 1,
> "_source": {
> "index": {
> "_index": "MyIndex",
> "_id": "2018-17136"
> 
> ```

This document have specified `_id` and not `id` like in the other document. It looks like this may be the index record of the bulk format, meaning that you are not constructing the bulk request correctly. I am not a .NET developer so will not be able to help with the client usage.

---

<div class="post-metadata">

**Author:** ![Moni\_Hazarika](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moni_hazarika/32/124158_2.png) [@Moni\_Hazarika](https://discuss.elastic.co/u/Moni_Hazarika)\
**Post date:** [November 5, 2024, 8:12am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-api-with-net-for-upsert/370003/9 "2024-11-05T08:12:09Z")

</div>

Thanks @Christian_Dahlqvist  
Yes I tried to google but did not find the latest C# .NET usage on how to make the bulk API call in the correct format. But I am sending only \_id from my code as you can see below

```auto
string documentId = ((ModelRecord)document).Id; //ModelRecord is the POCO which has Id field and the unique identifier 
var elasticsearchId = new Id(documentId);
bulkIndexOperations.Add(new { Index = new { _index = indexName, _id = elasticsearchId } });
bulkIndexOperations.Add(document);

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 5, 2024, 8:15am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-api-with-net-for-upsert/370003/10 "2024-11-05T08:15:50Z")

</div>

To me it looks like each of the `add` method calls add a new document and not a separate line to the bulk format.

`{ Index = new { _index = indexName, _id = elasticsearchId } }`

seems to correspond to

```auto
"_source": {
  "index": {
    "_index": "MyIndex",
    "_id": "2018-17136"
  }
}

```

Does the `add` method have any other parameters, e.g. index name, operation and document id?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [November 5, 2024, 8:28am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-api-with-net-for-upsert/370003/11 "2024-11-05T08:28:48Z")

</div>

I agree. In Java, you call only once the `add` method with both the header (`_id`, `_index`) and the body (`document`).

---

<div class="post-metadata">

**Author:** ![Moni\_Hazarika](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moni_hazarika/32/124158_2.png) [@Moni\_Hazarika](https://discuss.elastic.co/u/Moni_Hazarika)\
**Post date:** [November 5, 2024, 8:35am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-api-with-net-for-upsert/370003/12 "2024-11-05T08:35:01Z")

</div>

> [@Christian\_Dahlqvist](#):
>
> It looks like the `id` field is used for the document id.

Based on my understanding I formed the Add. I did not find any sample for .NET in documentation on its usage.  
If I remove the below line bulk API fails  
`bulkIndexOperations.Add(new { Index = new { _index = indexName, _id = documentId } });`  
where I thought when its Index its and insert and its like below its an update operation

```auto
bulkIndexOperations.Add(new { Update = new { _index = indexName, _id = documentId } });
bulkIndexOperations.Add(new { doc = document });

```

Here bulkIndexOperations is just a List of object(List bulkIndexOperations) so I don't know if there is a fixed format. I tried to have something to match below  
\_id is NOT mandatory ES will create one if not sent.

```auto
POST _bulk
{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1" }

```

I am already passing indexName and documentId right? what am I missing?

---

<div class="post-metadata">

**Author:** ![Moni\_Hazarika](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moni_hazarika/32/124158_2.png) [@Moni\_Hazarika](https://discuss.elastic.co/u/Moni_Hazarika)\
**Post date:** [November 5, 2024, 8:40am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-api-with-net-for-upsert/370003/13 "2024-11-05T08:40:06Z")

</div>

If I google .NET usage it takes me to this page - [Multiple documents with BulkAllObservable helper | Elasticsearch .NET Client [8.15] | Elastic](https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/bulkall-observable.html)  
[Indexing documents | Elasticsearch .NET Client [8.15] | Elastic](https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/indexing-documents.html)  
but says - This page has been deleted.

---

<div class="post-metadata">

**Author:** ![Moni\_Hazarika](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moni_hazarika/32/124158_2.png) [@Moni\_Hazarika](https://discuss.elastic.co/u/Moni_Hazarika)\
**Post date:** [November 5, 2024, 8:50am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-api-with-net-for-upsert/370003/14 "2024-11-05T08:50:08Z")

</div>

This is my understanding and I did not get any reference in the internet

```auto
// First add the metadata for the bulk operation (the "header")
    bulkIndexOperations.Add(new 
    { 
        Index = new 
        { 
            _index = indexName, 
            _id = documentId 
        } 
    });

    // Then add the document itself (the "body")
    bulkIndexOperations.Add(document);

```

So, the first line imo (`{ "index": { "_index": "MyIndex", "_id": "2018-17136" } }`) is the metadata, which specifies that the operation is an index operation and the target index and document ID.  
The second line (`{ "id": "2018-17136", "fieldA": true, "fieldB": false }`) is the actual document, which contains the fields to be indexed.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 5, 2024, 8:59am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-api-with-net-for-upsert/370003/15 "2024-11-05T08:59:35Z")

</div>

Based on what you see in Elasticsearch it seems clear that the client handles these two commands as separate documents and not two lines of the bulk request.

Have you looked at the tests available within the client repo to see how it is used?

---

<div class="post-metadata">

**Author:** ![Moni\_Hazarika](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moni_hazarika/32/124158_2.png) [@Moni\_Hazarika](https://discuss.elastic.co/u/Moni_Hazarika)\
**Post date:** [November 6, 2024, 6:32am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-api-with-net-for-upsert/370003/16 "2024-11-06T06:32:18Z")

</div>

Hi @Christian_Dahlqvist I looked at the github repo and felt this is relevant [elasticsearch-net/tests/Tests/Document/Multiple/Bulk/BulkApiTests.cs at main · elastic/elasticsearch-net · GitHub](https://github.com/elastic/elasticsearch-net/blob/main/tests/Tests/Document/Multiple/Bulk/BulkApiTests.cs)  
But did not completely get how exactly to form the correct bulk request.

For now I am continuing with below code but I dont know how to pass the operation type - as to its Insert of Update

```auto
string documentId = ((MyRecord)document).Id;
    //Check for duplicates in the bulk operations first
    bool alreadyExistsInBulk = bulkIndexOperations.Any(op =>
    {
      if (op is MyRecord record)
      {
        return record.Id == documentId; // Check if the document ID is already in the operations list
      }
      return false;
    });

    if (!alreadyExistsInBulk)
    {
      bulkIndexOperations.Add(document);
    }
if (bulkIndexOperations.Count >= batchSize)
{
   var bulkResponse = await _elasticsearchClient.BulkAsync(b => b
                    .Index(indexName)
                    .IndexMany(bulkIndexOperations));
}

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 6, 2024, 6:34am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-api-with-net-for-upsert/370003/17 "2024-11-06T06:34:22Z")

</div>

I am not a .NET developer and have therefore never used that client so will unfortunately not be able to help with this.