Performance Issue on Inserting Data

(Parthiban P) #1

We are using Elastic Search Nest client for our application , we did load testing testing on inserting documents into Elastic Search DB and we found that there is a performance issue.

Index Size 2Kb , Shard Count 5, replica 1, Single Instance DB(no cluster)

Final Results stats tat we could able to insert only 3 records per Second .. is this expected result?

when we increase client threads , results become 2 records per Second , kindly suggest

Test Case

5 parallel thread , each thread synchronously pushing records(10000) .it took around 50 minutes(each thread), 3 record per second insert.

I have created N (5 for this test case) number of Clients on the application startup and reusing it for all the requests.

Server Details :
16GB RAM , 64 bit OS, Intel Core i7 , Windows Server 2008 R2 , 500 GB hard disk

Client Machine:
16GB RAM , 64 bit OS, Intel Core i7, Windows 7 professional

(Christian Dahlqvist) #2

Are you using bulk requests? If so, is 10000 you mentioned your bulk size? Are you reusing the client and the persistent connections it creates across calls?

(Parthiban P) #3

Yes connections are persistent and reusing the client across calls.. In my case , i can't use Bulk API .. it is synchronous insert API call ..

(Mark Walkom) #4

Then use bulk, it'll help a lot.


I am currently doing POC using Nest on ES 5.0. I am getting much higher throughout with much lower spec machine.

I use indexmany method to bulk the requests and also using parallel foreach with 5-6 max parallel threads. Including the time involved in extracting data from a back end database, building the request and pushing it to ES, I am able to load around million documents in around 3.5-4 minutes. I am running this on a VM with 4GB RAM and 2 core Xeon allocation.

With your specs, if you start using indexmany, you should get much higher throughput. Indexmany will take in a list as input. So you will need to try out different batch sizes to identify what best works for your machine specs.

(Parthiban P) #6

In my use case i can't use BulkAPI , what would be expected result of Indexing? how many records can be inserted using Nest client in a sec ?

(Christian Dahlqvist) #7

Your indexing rate seems low even if indexing a single document per request. Do your document have a uniform structure, containing the same fields or are mappings likely to need to be updated frequently as well? What is the size of your documents?

(Parthiban P) #8

Yes , the document has uniform structure which doesn't required to update mapping . The document size is 2KB . Is the issue related to Nest which uses HttpWebRequest for each call ??


I would like to further understand your use case where you have 10000 documents to index but can't use bulk API. Why do you have that restriction?

(Parthiban P) #10

The business usecase is to insert a single data on a API call but i just did load testing to share the results.

Is there any other ways/options to improve throughput?

(system) #11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.