# Performance Issue on Inserting Data **URL:** \ **Category:** Elasticsearch\ **Created:** [December 15, 2016, 12:31pm UTC](https://discuss.elastic.co/t/performance-issue-on-inserting-data/69163 "2016-12-15T12:31:09Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![parthiban\_p](https://avatars.discourse-cdn.com/v4/letter/p/3da27b/32.png) [@parthiban\_p](https://discuss.elastic.co/u/parthiban_p)\ **Post date:** [December 15, 2016, 12:31pm UTC](https://discuss.elastic.co/t/performance-issue-on-inserting-data/69163/1 "2016-12-15T12:31:09Z")
We are using **Elastic Search Nest client** for our application , we did load testing testing on inserting documents into Elastic Search DB and we found that there is a performance issue. Index Size 2Kb , Shard Count 5, replica 1, Single Instance DB(no cluster) Final Results stats tat we could able to insert only 3 records per Second .. is this expected result? when we increase client threads , results become 2 records per Second , kindly suggest Test Case 5 parallel thread , each thread synchronously pushing records(10000) .it took around 50 minutes(each thread), 3 record per second insert. I have created N (5 for this test case) number of Clients on the application startup and reusing it for all the requests. Server Details : 16GB RAM , 64 bit OS, Intel Core i7 , Windows Server 2008 R2 , 500 GB hard disk Client Machine: 16GB RAM , 64 bit OS, Intel Core i7, Windows 7 professional ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 15, 2016, 12:35pm UTC](https://discuss.elastic.co/t/performance-issue-on-inserting-data/69163/2 "2016-12-15T12:35:26Z")
Are you using bulk requests? If so, is 10000 you mentioned your bulk size? Are you reusing the client and the persistent connections it creates across calls? ---
**Author:** ![parthiban\_p](https://avatars.discourse-cdn.com/v4/letter/p/3da27b/32.png) [@parthiban\_p](https://discuss.elastic.co/u/parthiban_p)\ **Post date:** [December 15, 2016, 12:39pm UTC](https://discuss.elastic.co/t/performance-issue-on-inserting-data/69163/3 "2016-12-15T12:39:27Z")
Yes connections are persistent and reusing the client across calls.. In my case , i can't use Bulk API .. it is synchronous insert API call .. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 15, 2016, 11:04pm UTC](https://discuss.elastic.co/t/performance-issue-on-inserting-data/69163/4 "2016-12-15T23:04:20Z")
Then use bulk, it'll help a lot. ---
**Author:** ![askids](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/askids/32/22252_2.png) [@askids](https://discuss.elastic.co/u/askids)\ **Post date:** [December 16, 2016, 1:22am UTC](https://discuss.elastic.co/t/performance-issue-on-inserting-data/69163/5 "2016-12-16T01:22:53Z")
I am currently doing POC using Nest on ES 5.0. I am getting much higher throughout with much lower spec machine. I use indexmany method to bulk the requests and also using parallel foreach with 5-6 max parallel threads. Including the time involved in extracting data from a back end database, building the request and pushing it to ES, I am able to load around million documents in around 3.5-4 minutes. I am running this on a VM with 4GB RAM and 2 core Xeon allocation. With your specs, if you start using indexmany, you should get much higher throughput. Indexmany will take in a list as input. So you will need to try out different batch sizes to identify what best works for your machine specs. ---
**Author:** ![parthiban\_p](https://avatars.discourse-cdn.com/v4/letter/p/3da27b/32.png) [@parthiban\_p](https://discuss.elastic.co/u/parthiban_p)\ **Post date:** [December 16, 2016, 6:09am UTC](https://discuss.elastic.co/t/performance-issue-on-inserting-data/69163/6 "2016-12-16T06:09:14Z")
In my use case i can't use BulkAPI , what would be expected result of Indexing? how many records can be inserted using Nest client in a sec ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 16, 2016, 6:13am UTC](https://discuss.elastic.co/t/performance-issue-on-inserting-data/69163/7 "2016-12-16T06:13:03Z")
Your indexing rate seems low even if indexing a single document per request. Do your document have a uniform structure, containing the same fields or are mappings likely to need to be updated frequently as well? What is the size of your documents? ---
**Author:** ![parthiban\_p](https://avatars.discourse-cdn.com/v4/letter/p/3da27b/32.png) [@parthiban\_p](https://discuss.elastic.co/u/parthiban_p)\ **Post date:** [December 16, 2016, 6:49am UTC](https://discuss.elastic.co/t/performance-issue-on-inserting-data/69163/8 "2016-12-16T06:49:03Z")
Yes , the document has uniform structure which doesn't required to update mapping . The document size is 2KB . Is the issue related to Nest which uses HttpWebRequest for each call ?? ---
**Author:** ![askids](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/askids/32/22252_2.png) [@askids](https://discuss.elastic.co/u/askids)\ **Post date:** [December 16, 2016, 6:59am UTC](https://discuss.elastic.co/t/performance-issue-on-inserting-data/69163/9 "2016-12-16T06:59:45Z")
I would like to further understand your use case where you have 10000 documents to index but can't use bulk API. Why do you have that restriction? ---
**Author:** ![parthiban\_p](https://avatars.discourse-cdn.com/v4/letter/p/3da27b/32.png) [@parthiban\_p](https://discuss.elastic.co/u/parthiban_p)\ **Post date:** [December 16, 2016, 7:04am UTC](https://discuss.elastic.co/t/performance-issue-on-inserting-data/69163/10 "2016-12-16T07:04:07Z")
The business usecase is to insert a single data on a API call but i just did load testing to share the results. Is there any other ways/options to improve throughput? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 13, 2017, 7:04am UTC](https://discuss.elastic.co/t/performance-issue-on-inserting-data/69163/11 "2017-01-13T07:04:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.