NEST indexing missing documents without errors

Elastic Stack Elasticsearch
1

Hi there,

So i'm trying to index around 636k documents into my ES node. I have a 1 node cluster where i end up with around 550k/636k documents, and i don't get any error messages at all while indexing.

Then i try on my local machine with 3 node setup and it misses documents aswell.

I'm using the NEST client, i make multiple SQL calls, combine the data into a document, gather 100 documents and bulk push them.

foreach (int a in ids)
                {
                    counter++;
                    Console.WriteLine("Running #: " + counter);
                    Declaration declaration = CreateDeclaration();
                    string id = a + ";";
                    declaration = _elasticRepository.GetDecQuery(Constants.DEC_SQL + id, declaration);
                    declaration = _elasticRepository.GetXMLQuery(Constants.XML_SQL + id, declaration);
                    declaration = _elasticRepository.GetSpecialsQuery(Constants.SPECIAL_SQL + id, declaration);
                    declaration = _elasticRepository.GetPrQuery(Constants.PR_SQL + id, declaration);
                    declaration = _elasticRepository.GetGoodsQuery(Constants.GOODS_SQL + id, declaration);
                    declaration = _elasticRepository.GetDocsQuery(Constants.DOCS_SQL + id, declaration);
                    declaration = _elasticRepository.GetCtrQuery(Constants.CTR_SQL + id, declaration);

                    decList.Add(declaration);
                    if (decList.Count > 99 || counter == ids.Count)
                    {
                        LoadToElastic(client, decList);
                        decList.Clear();
                        if(counter == ids.Count) { Log.Information("Done loading: " + DateTime.Now.ToString()); }
                    }

//LoadToElastic function
 var indexManyResponse = client.IndexMany(declaration);
            if (indexManyResponse.Errors)
            {
                 Log.Error("Logged errors in indexMany between dec_ids: " + declaration[0] + " - " + declaration[99]);
                 Console.WriteLine("Error occured in indexMany: ");
                    
            }
            if (!indexManyResponse.IsValid)
            {
                Log.Error("Logged an invalid response between dec_ids: " + declaration[0] + " - " + declaration[99]);
                Console.WriteLine("Error occured in indexmany");
            }

This is a general overview of the implementation, in neither responses i get an error. Any ideas where to look at?

2

I don't know nest, so hopefully one of the clients team will drop in. A few clarifying questions though;

  • Are you using the bulk helper?
  • Are you checking and logging the responses back from Elasticsearch?
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.