Performance Issue on Indexing

[parthiban_p]

We are using Elastic Search Nest client for our application , we did load testing testing on inserting documents into Elastic Search DB and we found that there is a performance issue.

Index Size 2Kb , Shard Count 5, replica 1, Single Instance DB(no cluster)

Final Results stats tat we could able to insert only 3 records per Second .. is this expected result?

when we increase client threads , results become 2 records per Second , kindly suggest ..

[cbuescher (Christoph)]

Hi,
what hardware are you running on? How do you test indexing performance, e.g. are you indexing each individual document or are you using the bulk apis? Any changes to the default configuration?

[Christian_Dahlqvist]

That sounds very slow even if you are indexing single events instead of using the significantly more efficient bulk requests. What size are your events? Are you initiating the client at the start of the benchmark and reusing it throughout?

[parthiban_p]

We are using Index API not the Bulk apis ..

Server Details :
16GB RAM , 64 bit OS, Intel Core i7 , Windows Server 2008 R2 , 500 GB hard disk

Client Machine:

16GB RAM , 64 bit OS, Intel Core i7, Windows 7 professional

Test Case

5 parallel thread , each thread synchronously pushing records(10000) .it took around 50 minutes(each thread), 3 record per second insert.

i am creating and deleting the client connection on every index request . not reusing the client connection.

[Christian_Dahlqvist]

That explains your poor performance. Creating clients is expensive, which is why all language clients are designed to be created and then reused in order to take advantage of persistent connections to the cluster. Using bulk requests will give you an additional boost to performance.