Performance Issue on Inserting Data

Elastic Stack Elasticsearch
1

We are using Elastic Search Nest client for our application , we did load testing testing on inserting documents into Elastic Search DB and we found that there is a performance issue.

Index Size 2Kb , Shard Count 5, replica 1, Single Instance DB(no cluster)

Final Results stats tat we could able to insert only 3 records per Second .. is this expected result?

when we increase client threads , results become 2 records per Second , kindly suggest

Test Case

5 parallel thread , each thread synchronously pushing records(10000) .it took around 50 minutes(each thread), 3 record per second insert.

I have created N (5 for this test case) number of Clients on the application startup and reusing it for all the requests.

Server Details :
16GB RAM , 64 bit OS, Intel Core i7 , Windows Server 2008 R2 , 500 GB hard disk

Client Machine:
16GB RAM , 64 bit OS, Intel Core i7, Windows 7 professional

2

Are you using bulk requests? If so, is 10000 you mentioned your bulk size? Are you reusing the client and the persistent connections it creates across calls?

3

Yes connections are persistent and reusing the client across calls.. In my case , i can't use Bulk API .. it is synchronous insert API call ..

4

Then use bulk, it'll help a lot.

5

I am currently doing POC using Nest on ES 5.0. I am getting much higher throughout with much lower spec machine.

I use indexmany method to bulk the requests and also using parallel foreach with 5-6 max parallel threads. Including the time involved in extracting data from a back end database, building the request and pushing it to ES, I am able to load around million documents in around 3.5-4 minutes. I am running this on a VM with 4GB RAM and 2 core Xeon allocation.

With your specs, if you start using indexmany, you should get much higher throughput. Indexmany will take in a list as input. So you will need to try out different batch sizes to identify what best works for your machine specs.

6

In my use case i can't use BulkAPI , what would be expected result of Indexing? how many records can be inserted using Nest client in a sec ?

7

Your indexing rate seems low even if indexing a single document per request. Do your document have a uniform structure, containing the same fields or are mappings likely to need to be updated frequently as well? What is the size of your documents?

8

Yes , the document has uniform structure which doesn't required to update mapping . The document size is 2KB . Is the issue related to Nest which uses HttpWebRequest for each call ??

9

I would like to further understand your use case where you have 10000 documents to index but can't use bulk API. Why do you have that restriction?

10

The business usecase is to insert a single data on a API call but i just did load testing to share the results.

Is there any other ways/options to improve throughput?

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.