# ES: Indexing Rate dropping significantly while Indexing Latency staying about the same **URL:** \ **Category:** Elasticsearch\ **Created:** [April 14, 2020, 4:20pm UTC](https://discuss.elastic.co/t/es-indexing-rate-dropping-significantly-while-indexing-latency-staying-about-the-same/227960 "2020-04-14T16:20:45Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![gurtgurt](https://avatars.discourse-cdn.com/v4/letter/g/7993a0/32.png) [@gurtgurt](https://discuss.elastic.co/u/gurtgurt)\ **Post date:** [April 14, 2020, 4:20pm UTC](https://discuss.elastic.co/t/es-indexing-rate-dropping-significantly-while-indexing-latency-staying-about-the-same/227960/1 "2020-04-14T16:20:45Z")
Hi, I'm indexing ~140 GB of data via the bulk API on a managed AWS instance. I'm noticing the Indexing rate is almost at 30% of what it started at, while the indexing latency is staying the same. ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/d/4d829d17dc43849c1b4082b2652d216760229b3b.png) I'm using EBS SSD as the backing store with 2 nodes with 64 gb memory each. My CPU and memory usage seem at pretty normal levels as well: ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/e/9e1411fd77fe03dff764fa3d56d4d52242cd708a.png) Any ideas on what could be throttling this? Does AWS maybe throttle inbound traffic or writes to EBS? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 14, 2020, 5:17pm UTC](https://discuss.elastic.co/t/es-indexing-rate-dropping-significantly-while-indexing-latency-staying-about-the-same/227960/2 "2020-04-14T17:17:47Z")
The larger shards get, the larger the segments get that need to be merged, which leads to more disk I/O. If you are using gp2 EBS, this has a fixed level of IOPS that you will be reaching once you have depleted you IOPS burst bucket. I am therefore not surprised you are seeing indexing rate drop over time unless you are using EBS with sufficient PIOPS. ---
**Author:** ![gurtgurt](https://avatars.discourse-cdn.com/v4/letter/g/7993a0/32.png) [@gurtgurt](https://discuss.elastic.co/u/gurtgurt)\ **Post date:** [April 14, 2020, 6:08pm UTC](https://discuss.elastic.co/t/es-indexing-rate-dropping-significantly-while-indexing-latency-staying-about-the-same/227960/3 "2020-04-14T18:08:57Z")
Gotcha, that makes sense. I found this little bit of info in the AWS docs > An important thing to note is that for any gp2 volume larger than 1 TiB, the baseline performance is greater than the burst performance. I'm currently using 500 GB gp2 volumes, I'm bumping it up to 1 TB to see if there is a significant improvement. Also for a large batch job like this where data isn't needed in real time, increasing the index.refresh\_interval should help as well right? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 14, 2020, 6:15pm UTC](https://discuss.elastic.co/t/es-indexing-rate-dropping-significantly-while-indexing-latency-staying-about-the-same/227960/4 "2020-04-14T18:15:07Z")
Yes, those should both help as allocated IOPS is proportional to volume size. Have a [look at the docs](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/tune-for-indexing-speed.html) for further suggestions. ---
**Author:** ![gurtgurt](https://avatars.discourse-cdn.com/v4/letter/g/7993a0/32.png) [@gurtgurt](https://discuss.elastic.co/u/gurtgurt)\ **Post date:** [April 14, 2020, 6:16pm UTC](https://discuss.elastic.co/t/es-indexing-rate-dropping-significantly-while-indexing-latency-staying-about-the-same/227960/5 "2020-04-14T18:16:56Z")
Thanks for the quick reply! Something I just thought of is if the issue is with the burst IOPS, shouldn't the indexing rate have a sharper drop, rather than a gradual drop? I thought of this while looking at this AWS documentation: [https://aws.amazon.com/blogs/database/understanding-burst-vs-baseline-performance-with-amazon-rds-and-gp2/](https://aws.amazon.com/blogs/database/understanding-burst-vs-baseline-performance-with-amazon-rds-and-gp2/) ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/3/9363a24b1b9a0268c00604192e86ed8d36725d78.png) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 14, 2020, 6:20pm UTC](https://discuss.elastic.co/t/es-indexing-rate-dropping-significantly-while-indexing-latency-staying-about-the-same/227960/6 "2020-04-14T18:20:40Z")
If you started indexing into an empty index the merging activity will also change over time so I am not sure that is the case. Do you have any monitoring of disk I/O and /or iowait over time? You shuld also be able to see from stats whether merging is being throttled or not. If you are specifying your own document IDs the indexing rate is also likely to drop as the index grows. ---
**Author:** ![gurtgurt](https://avatars.discourse-cdn.com/v4/letter/g/7993a0/32.png) [@gurtgurt](https://discuss.elastic.co/u/gurtgurt)\ **Post date:** [April 14, 2020, 6:34pm UTC](https://discuss.elastic.co/t/es-indexing-rate-dropping-significantly-while-indexing-latency-staying-about-the-same/227960/7 "2020-04-14T18:34:09Z")
Yes I started with indexing into an empty index, here are some more metrics from the beginning of the ingest: ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/5/059d63443cd929b1e1a3a5c78507d6f746a33251.png) We are in fact specifying our own document IDs, but we need to do this to avoid potential deduplication of our data in ElasticSearch. If there is a different way to deal with duplicate data, I'd be more than happy to switch to using ES doc IDs ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 14, 2020, 6:39pm UTC](https://discuss.elastic.co/t/es-indexing-rate-dropping-significantly-while-indexing-latency-staying-about-the-same/227960/8 "2020-04-14T18:39:06Z")
If you specify your own IDs, which generally as far as I know is required for deduplication, each insert is in fact a potenial update and Elasticsearch must check if the ID already exists. This will slow down indexing speed over time as the number of documents to check increases. If you can sort your documents by ID lexically before indexing you may see some boost (or at least slower deterioration of performance - see [this blog post](https://www.elastic.co/blog/efficient-duplicate-prevention-for-event-based-data-in-elasticsearch) for further details). Faster storage will also help. ---
**Author:** ![gurtgurt](https://avatars.discourse-cdn.com/v4/letter/g/7993a0/32.png) [@gurtgurt](https://discuss.elastic.co/u/gurtgurt)\ **Post date:** [April 15, 2020, 12:34am UTC](https://discuss.elastic.co/t/es-indexing-rate-dropping-significantly-while-indexing-latency-staying-about-the-same/227960/9 "2020-04-15T00:34:44Z")
Just as an update I retried the batch ingest with a 1 TB volume (so bursts shouldn't have an effect) and also with the following settings: "number\_of\_replicas" : 0, "refresh\_interval": -1 and we're seeing the same pattern of decrease in index rate. ---
**Author:** ![gurtgurt](https://avatars.discourse-cdn.com/v4/letter/g/7993a0/32.png) [@gurtgurt](https://discuss.elastic.co/u/gurtgurt)\ **Post date:** [April 15, 2020, 1:40am UTC](https://discuss.elastic.co/t/es-indexing-rate-dropping-significantly-while-indexing-latency-staying-about-the-same/227960/10 "2020-04-15T01:40:37Z")
Another update: Using the auto generated ID fixed the issue. Thanks for your help! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 13, 2020, 1:40am UTC](https://discuss.elastic.co/t/es-indexing-rate-dropping-significantly-while-indexing-latency-staying-about-the-same/227960/11 "2020-05-13T01:40:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.