# Slow bulk indexing with lots of different 'types' **URL:** \ **Category:** Elasticsearch\ **Created:** [October 17, 2016, 11:06am UTC](https://discuss.elastic.co/t/slow-bulk-indexing-with-lots-of-different-types/63185 "2016-10-17T11:06:44Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![saucam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saucam/32/11178_2.png) [@saucam](https://discuss.elastic.co/u/saucam)\ **Post date:** [October 17, 2016, 11:06am UTC](https://discuss.elastic.co/t/slow-bulk-indexing-with-lots-of-different-types/63185/1 "2016-10-17T11:06:44Z")
Hi all, I have been trying to index data into elasticsearch using bulk index api. I am using elastic4s to construct the bulk query and execute using the client which uses java api under the hood. I have a 5 node elastic cluster (16GB RAM, 8 cores each) I have set ES\_HEAP to 8G on all nodes. The data I have is very small, and I use custom routing when bulk indexing. The routing field is set to a city id on which I want to distribute the data. So I am trying to index many different documents pertaining to different cities. I also use the cityId as 'type' when indexing so when searching I can set the type to that particular cityId. Overall I have data for 60000 different cities. I club several documents for each city (10 - 12) in a single bulk call. If I club more, then the performance deteriorates even faster. The size of each call varies between 20 - 150 kb. I have a total of about 1 million documents to index. Each call is synchronous, so I wait for each bulk call to return. I use 3 parallel workers to push data to elastic. So each of them are doing similar calls to elastic at the same time. Most of the data is long / double with only 2 - 3 fields as string. I have also set refresh interval to -1 , memory.index\_buffer\_size to 40% , merge.scheduler.max\_thread\_count to 1 and store.throttle.type to "none". Also I have set replica for the index to 0 for indexing data. Right at the start I get an index speed of about 150 - 200 documents / sec which quickly deteriorates. Soon after about 10 - 15 min It falls to 40 - 50 documents / seconds and goes even below that sometimes. It picks up randomly to 100 / second in between but keeps falling back to 30 - 40 per sec. Also after 30 min I also see at one particular time the bulk queueu entirely full on all the five nodes, but it recovers from that after some time. What more can I do to get better performance than this ? Any pointers are highly appreciated. Best Regards Yash ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 17, 2016, 11:13am UTC](https://discuss.elastic.co/t/slow-bulk-indexing-with-lots-of-different-types/63185/2 "2016-10-17T11:13:24Z")
Try indexing your data using a single type. Having that many different types is a bad practice, and the mappings, and therefore the cluster state, will need to be updated frequently, which will cause bad performance, especially if this is a high cardinality field. ---
**Author:** ![saucam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saucam/32/11178_2.png) [@saucam](https://discuss.elastic.co/u/saucam)\ **Post date:** [October 17, 2016, 11:15am UTC](https://discuss.elastic.co/t/slow-bulk-indexing-with-lots-of-different-types/63185/3 "2016-10-17T11:15:20Z")
Hi , Thanks for the prompt response. Can I still use cityId as routing parameter, and just eliminate the type ? Then while querying I can atleast make use of routing field ? I have data for 60K unique cities btw. Best Regards Yash ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 17, 2016, 11:17am UTC](https://discuss.elastic.co/t/slow-bulk-indexing-with-lots-of-different-types/63185/4 "2016-10-17T11:17:21Z")
Yes, you can certainly still use routing. ---
**Author:** ![saucam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saucam/32/11178_2.png) [@saucam](https://discuss.elastic.co/u/saucam)\ **Post date:** [October 17, 2016, 11:20am UTC](https://discuss.elastic.co/t/slow-bulk-indexing-with-lots-of-different-types/63185/5 "2016-10-17T11:20:06Z")
Thank you ! Also any other parameters I should play with ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 17, 2016, 11:25am UTC](https://discuss.elastic.co/t/slow-bulk-indexing-with-lots-of-different-types/63185/6 "2016-10-17T11:25:49Z")
You should be able to use a larger bulk size - a good starting point might be around 500. ---
**Author:** ![saucam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saucam/32/11178_2.png) [@saucam](https://discuss.elastic.co/u/saucam)\ **Post date:** [October 17, 2016, 11:55am UTC](https://discuss.elastic.co/t/slow-bulk-indexing-with-lots-of-different-types/63185/7 "2016-10-17T11:55:56Z")
Thanks Christian, I definitely get a LOT better performance when using a single type. Will try better bulk size as well ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:11pm UTC](https://discuss.elastic.co/t/slow-bulk-indexing-with-lots-of-different-types/63185/8 "2017-07-05T22:11:45Z")