# 7 seconds to index document once i get close to 2million documents **URL:** \ **Category:** Elasticsearch\ **Created:** [March 4, 2018, 5:52pm UTC](https://discuss.elastic.co/t/7-seconds-to-index-document-once-i-get-close-to-2million-documents/122415 "2018-03-04T17:52:25Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![skydiver](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@skydiver](https://discuss.elastic.co/u/skydiver)\ **Post date:** [March 4, 2018, 5:52pm UTC](https://discuss.elastic.co/t/7-seconds-to-index-document-once-i-get-close-to-2million-documents/122415/1 "2018-03-04T17:52:25Z")
Hi I'm very new to elasticsearch and running as 3 node cluster using docker. I have a large amount of data around 250 million records which is currently stored between xml files and a mysql database and found that elasticsearch would be create to use for my needs. So i began following some video tutorials online, and manage to create/index 1,000,0000 fine, this was relatively quick too, using the localhost:9200 end point. I played around with a few queries and everything all ok. So i began to index (not sure if right term) a few more million records with the plan to do some more testing. However once i had around 1.6million documents indexed it seem to grind to a halt. Indexing a single document was/is taking around 7 seconds to do, where as before i could send 100 a second quiet easy. But the reading/queries seems fine. Im using raid5 with 3x8TB drives, binding the volume data to the raid drive instead of the default /var/lib/docker folder. I have done some reading but not 100% sure what i should be looking for. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 4, 2018, 9:29pm UTC](https://discuss.elastic.co/t/7-seconds-to-index-document-once-i-get-close-to-2million-documents/122415/2 "2018-03-04T21:29:09Z")
What is the average size of your documents? Are you using the bulk API? Are you using custom document IDs? ---
**Author:** ![skydiver](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@skydiver](https://discuss.elastic.co/u/skydiver)\ **Post date:** [March 4, 2018, 9:49pm UTC](https://discuss.elastic.co/t/7-seconds-to-index-document-once-i-get-close-to-2million-documents/122415/3 "2018-03-04T21:49:55Z")
the documents range from 5-16kb in size, they are an xml document and im just storing them as a string at the moment, as they are currently files on my hard drive. Im using a custom id for example "/customer/listing/182939403" the ID is represented in the document too. Im not using the bulk api at the moment, i wrote a crude script which looped through my existing data set, which was writing/indexing 157 per second on average. As soon as i noticed it slowed to a grinding halt i stop the script. I then went back to the Kibana Dev tools and indexed a single test document which just had a small amount of text. Its at this point i realised it was the indexing of the document which has slowed down massively. At first i wondered if it was my raid5 or something. But when i did a search query in kibana it was instant no lag at all. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 4, 2018, 10:47pm UTC](https://discuss.elastic.co/t/7-seconds-to-index-document-once-i-get-close-to-2million-documents/122415/4 "2018-03-04T22:47:09Z")
How large is the index? How many shards do you have? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 1, 2018, 10:47pm UTC](https://discuss.elastic.co/t/7-seconds-to-index-document-once-i-get-close-to-2million-documents/122415/5 "2018-04-01T22:47:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.