# Indexing speed slowdown **URL:** \ **Category:** Elasticsearch\ **Created:** [September 21, 2017, 8:43am UTC](https://discuss.elastic.co/t/indexing-speed-slowdown/101288 "2017-09-21T08:43:19Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![philv](https://avatars.discourse-cdn.com/v4/letter/p/ebca7d/32.png) [@philv](https://discuss.elastic.co/u/philv)\ **Post date:** [September 21, 2017, 8:43am UTC](https://discuss.elastic.co/t/indexing-speed-slowdown/101288/1 "2017-09-21T08:43:19Z")
I've noticed a slowdown in indexing speed going from esv 2.3 to 5.5.0. It's early days, I have no precise figures and there are a tonne of other things that may differ, but my first thought was that it may be a change I made to doc\_values- there are more doc\_values now. I presume this will slow indexing speed, but I wonder by how much. I'm seeing a doubling of the time at least to index my corpus of text. Any suggestions ? Thanks, Phil ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 21, 2017, 8:57am UTC](https://discuss.elastic.co/t/indexing-speed-slowdown/101288/2 "2017-09-21T08:57:03Z")
How large are your documents? How large bulk requests are you using? How many shards are you actively indexing into? ---
**Author:** ![philv](https://avatars.discourse-cdn.com/v4/letter/p/ebca7d/32.png) [@philv](https://discuss.elastic.co/u/philv)\ **Post date:** [September 21, 2017, 9:57am UTC](https://discuss.elastic.co/t/indexing-speed-slowdown/101288/3 "2017-09-21T09:57:26Z")
About 300 characters on average of text. Rest is unchanged from my previous settings in ES 2.3, which was: "number\_of\_replicas": 0, "refresh\_interval": "120s", "store.throttle.max\_bytes\_per\_sec" : "40mb", "index.translog.flush\_threshold\_size":"1g" ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 21, 2017, 10:42am UTC](https://discuss.elastic.co/t/indexing-speed-slowdown/101288/4 "2017-09-21T10:42:18Z")
What is the size of the bulk index requests you are sending to Elasticsearch (assuming you are using the bulk API)? ---
**Author:** ![philv](https://avatars.discourse-cdn.com/v4/letter/p/ebca7d/32.png) [@philv](https://discuss.elastic.co/u/philv)\ **Post date:** [September 21, 2017, 10:54am UTC](https://discuss.elastic.co/t/indexing-speed-slowdown/101288/5 "2017-09-21T10:54:46Z")
These are the parameters (again, unchanged) .setBulkSize( new ByteSizeValue(20, ByteSizeUnit.MB)) .setConcurrentRequests(calculateBulkLoadThreads() - 1) .build(); ---
**Author:** ![philv](https://avatars.discourse-cdn.com/v4/letter/p/ebca7d/32.png) [@philv](https://discuss.elastic.co/u/philv)\ **Post date:** [September 21, 2017, 10:56am UTC](https://discuss.elastic.co/t/indexing-speed-slowdown/101288/6 "2017-09-21T10:56:16Z")
And I observe a degradation over time. I start off getting tens of thousands of docs loaded per minute and after a while this rate drops ten fold. Stopping and restarting doesn't help and I suspect it's more to do with random IO or seek times as the index grows. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 21, 2017, 11:05am UTC](https://discuss.elastic.co/t/indexing-speed-slowdown/101288/7 "2017-09-21T11:05:38Z")
Are you indexing documents with a custom document ID or letting Elasticsearch assign the id? Do you update your documents? ---
**Author:** ![philv](https://avatars.discourse-cdn.com/v4/letter/p/ebca7d/32.png) [@philv](https://discuss.elastic.co/u/philv)\ **Post date:** [September 21, 2017, 1:21pm UTC](https://discuss.elastic.co/t/indexing-speed-slowdown/101288/8 "2017-09-21T13:21:07Z")
We use a custom ID. format: my-source\ __\__ \_\*\_1234\_1 where '12341678' is an integer and changes but the other stuff remains constant (in this specific example load). We don't update, this was a fresh load to a virgin index. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 21, 2017, 1:24pm UTC](https://discuss.elastic.co/t/indexing-speed-slowdown/101288/9 "2017-09-21T13:24:59Z")
If you do not need to update documents or use the id to ensure you don't get duplicates, letting Elasticsearch assign the document id will give better performance as Elasticsearch knows the documents do not already exist. You can read about the impact of id formats on indexing speed in [this blog post](http://blog.mikemccandless.com/2014/05/choosing-fast-unique-identifier-uuid.html). ---
**Author:** ![philv](https://avatars.discourse-cdn.com/v4/letter/p/ebca7d/32.png) [@philv](https://discuss.elastic.co/u/philv)\ **Post date:** [September 21, 2017, 1:58pm UTC](https://discuss.elastic.co/t/indexing-speed-slowdown/101288/10 "2017-09-21T13:58:14Z")
That's just it- we need the ID to ensure we don't get duplicates ---
**Author:** ![philv](https://avatars.discourse-cdn.com/v4/letter/p/ebca7d/32.png) [@philv](https://discuss.elastic.co/u/philv)\ **Post date:** [September 21, 2017, 1:59pm UTC](https://discuss.elastic.co/t/indexing-speed-slowdown/101288/11 "2017-09-21T13:59:53Z")
and the thing is, it's the differential between index speed on 2.3 and 5.5.0 that interests me. The ID style hasn't changed. But, if there's nothing crazy about my process, that's cool, I can go investigate more deeply. ---
**Author:** ![philv](https://avatars.discourse-cdn.com/v4/letter/p/ebca7d/32.png) [@philv](https://discuss.elastic.co/u/philv)\ **Post date:** [September 21, 2017, 3:24pm UTC](https://discuss.elastic.co/t/indexing-speed-slowdown/101288/12 "2017-09-21T15:24:42Z")
bah, its nothing to do with ES. Sorry to have wasted your time. Found a different cause for slowness. Face-palm. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 19, 2017, 3:24pm UTC](https://discuss.elastic.co/t/indexing-speed-slowdown/101288/13 "2017-10-19T15:24:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.