# Speeding up indexing in ES 2.2.0 **URL:** \ **Category:** Elasticsearch\ **Created:** [March 28, 2016, 6:44pm UTC](https://discuss.elastic.co/t/speeding-up-indexing-in-es-2-2-0/45620 "2016-03-28T18:44:07Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![mkelkar](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@mkelkar](https://discuss.elastic.co/u/mkelkar)\ **Post date:** [March 28, 2016, 6:44pm UTC](https://discuss.elastic.co/t/speeding-up-indexing-in-es-2-2-0/45620/1 "2016-03-28T18:44:07Z")
Hi All, I am trying to test performance of service with ES 2.2.0 and I am seeing slowness in rate of indexing. We use SSDs for all of our data nodes. What I am seeing is that if I test with the same setup with ES 1.7.3, we can index documents on avg 70K/sec, but with ES 2.2.0, the rate drops to 40K/sec. What I am looking for is advice on what things I can start to look at to get better indexing performance. Here is a description of our test setup, which is identical for both 1.7.3 and 2.2.0 service nodes 3 m2.4xls ES master 3 nodes m3.2xlarge ES Data 9 c3.8xls We are using the default merge policy which comes with ES 2.2.0. For 1.7.3, we have set indices.store.throttle.type = none to speed up indexing. This setting (and various others) have been removed by ES 2.X. Any clues on what I can start looking into? Thanks, Madhav. ---
**Author:** ![tinle](https://avatars.discourse-cdn.com/v4/letter/t/c77e96/32.png) [@tinle](https://discuss.elastic.co/u/tinle)\ **Post date:** [March 28, 2016, 7:01pm UTC](https://discuss.elastic.co/t/speeding-up-indexing-in-es-2-2-0/45620/2 "2016-03-28T19:01:47Z")
The default translog flush policy in ES 2.x changed to synchronous. So there is a performance hit. If you want to use same policy as ES 1.X, set this in your ES config ``` # riskier, but faster (same as ES pre-2.x; default in 2.x is 'request') index.translog.durability: async ``` ---
**Author:** ![mkelkar](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@mkelkar](https://discuss.elastic.co/u/mkelkar)\ **Post date:** [March 28, 2016, 8:35pm UTC](https://discuss.elastic.co/t/speeding-up-indexing-in-es-2-2-0/45620/3 "2016-03-28T20:35:31Z")
Thanks Tinle for the pointer! I just tried it, and it has increased indexing speed by a little, but not by much...its still around 42K/sec mark..is there something else I can check? Thanks, Madhav. ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [March 28, 2016, 8:32pm UTC](https://discuss.elastic.co/t/speeding-up-indexing-in-es-2-2-0/45620/4 "2016-03-28T20:32:47Z")
You _should_ be able to use a larger bulk size to get performance to the point where the default is only a little slower than `async`. So, yeah, my suggestion is to try larger bulk sizes in 2.x and to make sure your refresh interval is high-ish, like 30 seconds. ---
**Author:** ![rusty](https://avatars.discourse-cdn.com/v4/letter/r/f17d59/32.png) [@rusty](https://discuss.elastic.co/u/rusty)\ **Post date:** [March 28, 2016, 9:34pm UTC](https://discuss.elastic.co/t/speeding-up-indexing-in-es-2-2-0/45620/5 "2016-03-28T21:34:57Z")
> [@mkelkar](#): > > indices.store.throttle.type = none This setting was moved to index template, so you can use it there. ``` "settings": { "index": { "store": { "throttle": { "type": "none" } } } } ``` You can also try to disable `doc_values` for non analyzed fields (if you don't really need this). This can also improve indexing rate and minimize disk usage. ---
**Author:** ![mkelkar](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@mkelkar](https://discuss.elastic.co/u/mkelkar)\ **Post date:** [March 28, 2016, 10:36pm UTC](https://discuss.elastic.co/t/speeding-up-indexing-in-es-2-2-0/45620/6 "2016-03-28T22:36:07Z")
Thanks Nik & Rusty! I will try these out and see what happens. @Rusty - is this new setting dynamic? I do not see it documented anywhere... Thanks, Madhav. ---
**Author:** ![rusty](https://avatars.discourse-cdn.com/v4/letter/r/f17d59/32.png) [@rusty](https://discuss.elastic.co/u/rusty)\ **Post date:** [March 29, 2016, 6:58am UTC](https://discuss.elastic.co/t/speeding-up-indexing-in-es-2-2-0/45620/7 "2016-03-29T06:58:46Z")
My bad, this setting [removed](https://www.elastic.co/guide/en/elasticsearch/reference/2.2/breaking_20_setting_changes.html#_merge_and_merge_throttling_settings) from ES 2.2 ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [March 29, 2016, 7:16am UTC](https://discuss.elastic.co/t/speeding-up-indexing-in-es-2-2-0/45620/8 "2016-03-29T07:16:38Z")
70k/sec - do you mean 70k docs per second, or 70 kilobytes per second? ---
**Author:** ![JoarSvensson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joarsvensson/32/8740_2.png) [@JoarSvensson](https://discuss.elastic.co/u/JoarSvensson)\ **Post date:** [March 29, 2016, 11:06am UTC](https://discuss.elastic.co/t/speeding-up-indexing-in-es-2-2-0/45620/9 "2016-03-29T11:06:02Z")
Like @rusty I would recommend trying to disable doc\_values for fields where applicable, to see if you experience a performance increase. More on doc\_values: [http://stackoverflow.com/questions/32332487/what-are-the-disadvantages-of-elasticsearch-doc-values](http://stackoverflow.com/questions/32332487/what-are-the-disadvantages-of-elasticsearch-doc-values) ---
**Author:** ![mkelkar](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@mkelkar](https://discuss.elastic.co/u/mkelkar)\ **Post date:** [March 29, 2016, 2:18pm UTC](https://discuss.elastic.co/t/speeding-up-indexing-in-es-2-2-0/45620/10 "2016-03-29T14:18:53Z")
@jprante its 70k docs per second. @JoarSvensson we have already disabled doc\_values where ever we could. I am testing out increased batch sizes as we speak, I will post the results soon enough. Thanks, Madhav. ---
**Author:** ![JoarSvensson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joarsvensson/32/8740_2.png) [@JoarSvensson](https://discuss.elastic.co/u/JoarSvensson)\ **Post date:** [April 4, 2016, 10:59am UTC](https://discuss.elastic.co/t/speeding-up-indexing-in-es-2-2-0/45620/11 "2016-04-04T10:59:17Z")
@mkelkar Any luck with increased batch sizes? ---
**Author:** ![mkelkar](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@mkelkar](https://discuss.elastic.co/u/mkelkar)\ **Post date:** [April 5, 2016, 10:57pm UTC](https://discuss.elastic.co/t/speeding-up-indexing-in-es-2-2-0/45620/12 "2016-04-05T22:57:09Z")
@JoarSvensson - I just finished doing a couple of large scale tests minutes ago... 1. Increased batch size for bulk-index-request to 5k instead of earlier 1k - this was total failure. ES started throwing out a bunch of exceptions, all of which had this - 'NotSerializableExceptionWrapper[Failed to acknowledge mapping update within [30s]' 2. I decreased batch size to 4k - similar story..ES did not throw out exceptions, but the bulk write rate was about 20k which was way lower as compared 40K to using batch size 1k ... Also, I tried to disable auto throttling on merges(index. and setting max\_thread\_count to 1 which did not help either... Its worth mentioning that our process runs in two phases , first phase is indexing heavy, and second phase is query heavy. The way it works is it 1. first indexes all documents in ES for a client ( phase1) . 2. as soon as all docs are written for a client, then it moves to Phase2 where the docs are updated according to our business rules and are indexed again.. What I have seen with 2.2.0 is that while phase1 is running for some clients, phase2 begins processing way faster than 1.7.3. I hunch is that in 1.7.3 because we set indices.throttle.type = none, indexing is topmost priority so we finish phase1 lot quicker, and Phase2 processing is automatically slowed down while phase1 is running...But for 2.2.0, Phase2 is way faster for some reason ( most probably because indexing is not the topmost priority anymore, and our queries are faster because of ES 2.2.0 optimizations ) ..... I would like to do something similar to 1.7.3 where we used to assign top most priority to indexing, but I am not aware of any settings which would do that in ES 2.2.0...I have already tried (index.merge.scheduler.auto\_throttle = false) and (index.merge.scheduler.max\_thread\_count=1) with no luck....any clues on how to proceed further? Thanks, Madhav. ---
**Author:** ![JoarSvensson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joarsvensson/32/8740_2.png) [@JoarSvensson](https://discuss.elastic.co/u/JoarSvensson)\ **Post date:** [April 6, 2016, 5:41am UTC](https://discuss.elastic.co/t/speeding-up-indexing-in-es-2-2-0/45620/13 "2016-04-06T05:41:06Z")
Interesting case indeed. I haven't come across settings to alter priority as of yet anyway. It feels like logic part of the internals of ES. The two thing on top of my mind is if you are either able to perform just one indexing and do the phase2 logic in one combined step prior to indexing. Or if you could add more nodes to help offload the cluster, or even have to separate clusters if possible to be sure you're not doing heavy indexing and querying at the same time. Hopefully someone else has better ideas. ---
**Author:** ![mkelkar](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@mkelkar](https://discuss.elastic.co/u/mkelkar)\ **Post date:** [April 6, 2016, 2:17pm UTC](https://discuss.elastic.co/t/speeding-up-indexing-in-es-2-2-0/45620/14 "2016-04-06T14:17:32Z")
We cannot perform phase2 logic within phase1 because it depends on all documents being written in ES first..we denormalize documents in Phase2, so if not all documents are available then our results would be incorrect. We already have a pair of clusters because we did see this conflict between indexing vs querying at the same time. We use one cluster just to transform our docs, and the second one just serves query traffic. However, because we have to do our doc processing in Phases, we have the same conflict on our doc processing cluster with 2.2.0...with 1.X we do not have any problems whatsoever.... I was planning to throttle our phase2 processing until Phase1 completes, but I wanted to see if there are better ideas for increasing priority for indexing.... Thanks, Madhav. ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [April 6, 2016, 9:02pm UTC](https://discuss.elastic.co/t/speeding-up-indexing-in-es-2-2-0/45620/15 "2016-04-06T21:02:14Z")
> [@mkelkar](#): > > 1. first indexes all documents in ES for a client ( phase1) .2. as soon as all docs are written for a client, then it moves to Phase2 where the docs are updated according to our business rules and are indexed again.. ES2 limits resources for a) bulk and b) segment merging (threads). There are good reasons for that, and there are no knobs to turn to enable something like "priority to indexing", because in ES1, you could only allocate more threads for bulk or segment merge than JVM can handle, which had serious side effects. I have a similar workload case like yours. You can avoid extra load when you create a new index in phase 2. Indexing new documents into an existing index (replacing existing docs) is more expensive than writing into an empty index. After that, switch an index alias form old to new index. ---
**Author:** ![mkelkar](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@mkelkar](https://discuss.elastic.co/u/mkelkar)\ **Post date:** [April 8, 2016, 5:51pm UTC](https://discuss.elastic.co/t/speeding-up-indexing-in-es-2-2-0/45620/16 "2016-04-08T17:51:17Z")
@jprante During phase2 I have no problems, its Phase1 that cause problems... Also, If I switch to bulk size of 500 docs / sec, I get a flood of these exceptions in the logs - java.util.concurrent.TimeoutException: Failed to acknowledge mapping update within [30s] at org.elasticsearch.cluster.action.index.MappingUpdatedAction.updateMappingOnMasterSynchronously(MappingUpdatedAction.java:122) at org.elasticsearch.cluster.action.index.MappingUpdatedAction.updateMappingOnMasterSynchronously(MappingUpdatedAction.java:112) at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:228) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:326) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:119) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68) at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:595) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:263) at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:260) at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) looks like it takes more than 30 seconds to ack the mapping update request...what should I start looking at changing? Thanks, Madhav. ---
**Author:** ![mkelkar](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@mkelkar](https://discuss.elastic.co/u/mkelkar)\ **Post date:** [May 4, 2016, 5:31pm UTC](https://discuss.elastic.co/t/speeding-up-indexing-in-es-2-2-0/45620/17 "2016-05-04T17:31:07Z")
FWIW, here is what I found - We used to have an ES node client embedded in our service JVM. During bulk indexing, our service was doing lot of work, which caused the embedded node client not to respond to cluster state updates...switching to transport client fixed the problem... ---
**Author:** ![Bruce\_Ritchie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bruce_ritchie/32/9370_2.png) [@Bruce\_Ritchie](https://discuss.elastic.co/u/Bruce_Ritchie)\ **Post date:** [May 4, 2016, 5:47pm UTC](https://discuss.elastic.co/t/speeding-up-indexing-in-es-2-2-0/45620/18 "2016-05-04T17:47:11Z")
The docs say that index.store.throttle.type setting was removed, but I still see it in IndexStore class and in [https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing-performance.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing-performance.html) documentation. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:53pm UTC](https://discuss.elastic.co/t/speeding-up-indexing-in-es-2-2-0/45620/19 "2017-07-05T22:53:57Z")