What is the correct way to insert document in elastic search using multiple documents using multiple threads in c#

mreddy9 · September 25, 2023, 7:39am

Hi All,

I'm using multiple threads in c# to isert documents in c# as below but some times document is not inserted into elstic search search.

var indexResponse = elasticClient.IndexDocument(testReport);

But if I try the following code which iterates through all documents in mongo db and then inserts into elastic all documents are getting inserted successfully. Please help to resolve this issue

await testcoll.Find(new BsonDocument()).ForEachAsync(async document =>
{
var indexResponse = elasticClient.IndexDocument(testReport);
})

dadoonet · September 25, 2023, 9:27am

Welcome!

How do you know that it's not inserted?
Are you running a GET by ID request? A Search? Are you getting an error?

A common mistake is to forget to run a refresh call after the document has been inserted. It's happening most of the time in integration tests.

For example, have a look at this (Java) code:

github.com

dadoonet/elasticsearch-java-client-demo/blob/cd89c87ac80b1583f59085bc1d849706c79b2a2b/src/test/java/fr/pilato/test/elasticsearch/hlclient/EsClientIT.java#L217-L225


      
          void createData() throws IOException {
              try {
                  client.indices().delete(dir -> dir.index("test"));
              } catch (ElasticsearchException ignored) { }
              client.index(ir -> ir.index("test").id("1").withJson(new StringReader("{\"foo\":\"bar\"}")));
              client.indices().refresh(rr -> rr.index("test"));
              SearchResponse<Void> response = client.search(sr -> sr.index("test"), Void.class);
              logger.info("response.hits.total.value = {}", response.hits().total().value());
          }

Note the refresh part:

client.indices().refresh(rr -> rr.index("test"));

mreddy9 · September 28, 2023, 12:22pm

Will it not delay the indexing process if we refresh the index for thousands of documents

dadoonet · September 28, 2023, 1:24pm

It will add some delay, yes. But you did not answer any of my questions...

How do you know that it's not inserted?
Are you running a GET by ID request?
A Search?
Are you getting an error?

mreddy9 · September 28, 2023, 4:56pm

Yes to know if it is not inserted I'm getting by ID.

dadoonet · September 28, 2023, 9:41pm

GET by id does not need a refresh. See Get API | Elasticsearch Guide [8.10] | Elastic

system · October 26, 2023, 9:41pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.