# Elasticsearch 7.6 is underperforming 6.8 **URL:** \ **Category:** Elasticsearch\ **Created:** [August 7, 2020, 6:26pm UTC](https://discuss.elastic.co/t/elasticsearch-7-6-is-underperforming-6-8/244211 "2020-08-07T18:26:53Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![danksim](https://avatars.discourse-cdn.com/v4/letter/d/a88e57/32.png) [@danksim](https://discuss.elastic.co/u/danksim)\ **Post date:** [August 7, 2020, 6:26pm UTC](https://discuss.elastic.co/t/elasticsearch-7-6-is-underperforming-6-8/244211/1 "2020-08-07T18:26:54Z")
Hello, I upgraded `Elastic Stack` from 6.8.2 to 7.6.2 (currently both clusters are running on `EKS` with `kubernetes` nodes on `AWS` `m5.4xlarge` machines `16CPUs` `64GB` - these are NOT `SSD`s) and I am experiencing degraded performance on 7.6.2. ```auto 6.8.2 setup: | 7.6.2 setup: | nodes: | client: | coordinator: replicas: 4 | replicas: 6 jvm heap: 2gb | jvm heap: 1gb cpu: 1 | cpu: 1 mem: 4gb | mem: 2gb data: | data: replicas: 8 | replicas: 8 jvm heap: 8gb | jvm heap: 8gb cpu: 1 | cpu: 6 mem: 16gb | mem: 16gb master: | master: replicas: 3 | replicas: 3 jvm heap: 1gb | jvm heap: 4gb cpu: 1 | cpu: 2 mem: 2gb | mem: 8gb | indices: | number_of_shards: | 2 | 6 number_of_replicas: | 1 | 1 refresh_interval: | 30s | 60s active_primary_shards: | 12241 | 1591 active_shards: | 24482 | 3182 ``` 6.8.2 has daily indices so it has way more shards. 7.6.2 does not have time-based indices but has a Rollover strategy using time-series indices with the zero-padded suffix`-n` (i.e. `fluentd--000001`) and uses the `Rollover API` every hour and rolls over hot indices if 7gb or larger and shrinking warm indices. (Note: I am not using `ILM` or `Curator` since I am using dynamic index naming) Everything else seems to be the same: - `write.queue_size`s are 200 - `indices.memory.index_buffer_size`s are 50% Although 6.8.2 has way more data (since 7.6.2 is fairly new), it is performing better. They are even receiving the same amount of data and both are getting a ton of 429s but 6.8.2 has no lag whereas 7.6.2 does. (according to `topk(50, kafka_consumergroup_group_max_lag_seconds{group=~"logstash-es"})`) `rate(elasticsearch_indices_indexing_index_total{cluster="$cluster",name=~"$name"}[$interval])` shows that 99% of the time, the `data` nodes are `indexing`! I have tried: - more CPU (though CPU usage avg is low) - more memory (though memory usage avg is low) - more JVM heap (though JVM heap usage avg is low) I have not tried: - more `Logstash` consumers (since the `write` queues were full, it didn't make sense) - `write.queue_size` to 1000 (as this will only take care of 429s) - more `data` nodes (I was going to try this soon) What could I be missing? ---
**Author:** ![Steve\_Mushero](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_mushero/32/22441_2.png) [@Steve\_Mushero](https://discuss.elastic.co/u/Steve_Mushero)\ **Post date:** [August 8, 2020, 5:24am UTC](https://discuss.elastic.co/t/elasticsearch-7-6-is-underperforming-6-8/244211/2 "2020-08-08T05:24:52Z")
Seems hard to compare if data and use is quite different between clusters, but why the low heap on coordinators as that would seem important to managing queries & results? No high CPU, high heap or long GC on any of them? Disk IO queues, as not SSD? Thread pool show queue? ---
**Author:** ![danksim](https://avatars.discourse-cdn.com/v4/letter/d/a88e57/32.png) [@danksim](https://discuss.elastic.co/u/danksim)\ **Post date:** [August 9, 2020, 10:24am UTC](https://discuss.elastic.co/t/elasticsearch-7-6-is-underperforming-6-8/244211/3 "2020-08-09T10:24:10Z")
> [@Steve\_Mushero](#): > > Seems hard to compare if data and use is quite different between clusters, but why the low heap on coordinators as that would seem important to managing queries & results? I can up the heap on the coordinator nodes and see if things improve but the heap usage is pretty low. Averages between 45-80%. Currently around 55%. **EDIT: My apologies. Looks like I made a mistake in my OP. The heap for the coordinator nodes are set at 4gb with 8gb of memory. I cannot edit the OP.** > [@Steve\_Mushero](#): > > No high CPU, high heap or long GC on any of them? Disk IO queues, as not SSD? No high CPU or heap. GC time shows an average of 10ms on all data nodes. Disk usage is low. I am not sure how to check disk IO. > [@Steve\_Mushero](#): > > Thread pool show queue? Thread pool shows there are queues for every data node at all times. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 9, 2020, 5:53pm UTC](https://discuss.elastic.co/t/elasticsearch-7-6-is-underperforming-6-8/244211/4 "2020-08-09T17:53:26Z")
What type and size of storage are you using? Indexing can be very I/O intensive so disk performance can quickly become a bottleneck. Check logs and cluster stats for evidence of merge throttling, which can indicate your storage is too slow and not able to keep up. You might also be able to run `íostat -x` on the nodes and see what that tells you about disk utilization and iowait. ---
**Author:** ![danksim](https://avatars.discourse-cdn.com/v4/letter/d/a88e57/32.png) [@danksim](https://discuss.elastic.co/u/danksim)\ **Post date:** [August 10, 2020, 5:19am UTC](https://discuss.elastic.co/t/elasticsearch-7-6-is-underperforming-6-8/244211/5 "2020-08-10T05:19:12Z")
> [@Christian\_Dahlqvist](#): > > What type and size of storage are you using? Indexing can be very I/O intensive so disk performance can quickly become a bottleneck. Both clusters (6.8.2 and 7.6.2) are currently using `AWS`'s `Throughput Optimized HDD ( "st1" ) volumes` specifically the `aws-ebs-st1`s. And I am witnessing no problems for 6.8.2. > [@Christian\_Dahlqvist](#): > > Check logs and cluster stats for evidence of merge throttling, which can indicate your storage is too slow and not able to keep up. You might also be able to run `íostat -x` on the nodes and see what that tells you about disk utilization and iowait. Looks like `iostats` is not installed in the official ES image (`Centos`). But I was able to pull up `io_stats` for one of the 8 `data` nodes from `/_nodes/stats`: ```auto "io_stats" : { "devices" : [ { "device_name" : "nvme1n1", "operations" : 72648455, "read_operations" : 1996898, "write_operations" : 70651557, "read_kilobytes" : 173918156, "write_kilobytes" : 2032345696 } ], "total" : { "operations" : 72648455, "read_operations" : 1996898, "write_operations" : 70651557, "read_kilobytes" : 173918156, "write_kilobytes" : 2032345696 } } } ``` Nothing jumps out at me. Am I missing something? Or does this look normal? ---
**Author:** ![Steve\_Mushero](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_mushero/32/22441_2.png) [@Steve\_Mushero](https://discuss.elastic.co/u/Steve_Mushero)\ **Post date:** [August 10, 2020, 7:31am UTC](https://discuss.elastic.co/t/elasticsearch-7-6-is-underperforming-6-8/244211/6 "2020-08-10T07:31:13Z")
If you are on HDD (spinning metal things), that's likely the bottleneck, though I'd think you'd see in both versions. You'll likely see iostat IO Queue, and merge throttling via: GET /\_node/stats - under merges, throttle time will increase, I assume (see total\_throttled\_time\_in\_millis) also index.is\_throttled Can't tell much from the ES stats, need that íostat -x output - you can install it easily with yum (and should be there, but maybe not in super minimal install). yum install sysstat I usually use: iostat -xdk 2 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 10, 2020, 7:38am UTC](https://discuss.elastic.co/t/elasticsearch-7-6-is-underperforming-6-8/244211/7 "2020-08-10T07:38:59Z")
There may be changes in disk I/O between the versions you are comparing, but I could not put my finger on it. Given the low I/I performance of the storage you are using I guess even a small change may make a big difference assuming indexing and query loads and patterns are the same. I would recommend upgrading to faster storage. ---
**Author:** ![Steve\_Mushero](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_mushero/32/22441_2.png) [@Steve\_Mushero](https://discuss.elastic.co/u/Steve_Mushero)\ **Post date:** [August 10, 2020, 8:26am UTC](https://discuss.elastic.co/t/elasticsearch-7-6-is-underperforming-6-8/244211/8 "2020-08-10T08:26:41Z")
Agreed faster disks ideal, though I imagine cost matters (you can also add more disks, as ES will use multiple data paths) - same major OS version, e.g. Centos 6 vs. 7 across your K8S? Else this gets into the infinite fun of disk stacks, IO tuning, etc. that have mostly gone away in the world of SSDs, but your iostat output still helpful. Wasn't there a flush or translog setting change V6 to 7? Something that might generate more fsyncs(), deadly to HDDs. Via translog.durability? Defaults seem same but you might check them and try async just to see if this is a key bottleneck (be sure you understand risk): [https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-translog.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-translog.html) For AWS, make sure EBS optimized if that works on these disk types; also are these on the same K8S cluster, or different, i.e. any cluster differences, and of course any resource sharing with other things. Would seem adding RAM is easy to try, but the node level IO info most useful but hard to do much about, as not a lot of knobs. ---
**Author:** ![danksim](https://avatars.discourse-cdn.com/v4/letter/d/a88e57/32.png) [@danksim](https://discuss.elastic.co/u/danksim)\ **Post date:** [August 10, 2020, 12:59pm UTC](https://discuss.elastic.co/t/elasticsearch-7-6-is-underperforming-6-8/244211/9 "2020-08-10T12:59:58Z")
> [@Steve\_Mushero](#): > > GET /\_node/stats - under merges, throttle time will increase, I assume (see total\_throttled\_time\_in\_millis) also index.is\_throttled Just looked this up for both clusters: ```auto 6.8.2: "merges" : { "current" : 0, "current_docs" : 0, "current_size_in_bytes" : 0, "total" : 1951274, "total_time_in_millis" : 1914707083, "total_docs" : 17775217277, "total_size_in_bytes" : 9534469482648, "total_stopped_time_in_millis" : 6173311, "total_throttled_time_in_millis" : 391270437, "total_auto_throttle_in_bytes" : 204626466137 }, ``` ```auto 7.6.2: "merges" : { "current" : 0, "current_docs" : 0, "current_size_in_bytes" : 0, "total" : 144392, "total_time_in_millis" : 84154676, "total_docs" : 1193124340, "total_size_in_bytes" : 656514864909, "total_stopped_time_in_millis" : 0, "total_throttled_time_in_millis" : 8862331, "total_auto_throttle_in_bytes" : 71011081830 } ``` Not sure if these numbers are meaningful since the two ES clusters have been running for a different number of days. Also, I've tried `yum install sysstat` but I got: ```auto [elasticsearch@elastic-stack-coordinator-0 ~]$ yum install sysstat Loaded plugins: fastestmirror, ovl ovl: Error while doing RPMdb copy-up: [Errno 13] Permission denied: '/var/lib/rpm/.dbenv.lock' You need to be root to perform this command. ``` Looks like I'll have to create a new image with `sysstats`... > [@Steve\_Mushero](#): > > though I imagine cost matters Yes. It matters! But I can talk to the team and try to switch to `SSD`s. > [@Steve\_Mushero](#): > > also are these on the same K8S cluster, or different, i.e. any cluster differences, and of course any resource sharing with other things. They are both running on the same `AWS EKS` cluster with more than enough `RAM`. Thank you all for helping! ---
**Author:** ![danksim](https://avatars.discourse-cdn.com/v4/letter/d/a88e57/32.png) [@danksim](https://discuss.elastic.co/u/danksim)\ **Post date:** [August 10, 2020, 1:04pm UTC](https://discuss.elastic.co/t/elasticsearch-7-6-is-underperforming-6-8/244211/10 "2020-08-10T13:04:05Z")
For the first few weeks 7.6.2 outperformed 6.8.2 (and everyone was happy). Not sure what has changed since then but it is now underperforming. I looked at everything I could think of and even restarted from scratch multiple times. I will try to upgrade to `SSD`s but it's probably a long shot given how much money we already spend on `AWS`. Thanks! ---
**Author:** ![danksim](https://avatars.discourse-cdn.com/v4/letter/d/a88e57/32.png) [@danksim](https://discuss.elastic.co/u/danksim)\ **Post date:** [August 10, 2020, 1:58pm UTC](https://discuss.elastic.co/t/elasticsearch-7-6-is-underperforming-6-8/244211/11 "2020-08-10T13:58:10Z")
I've noticed that most of the `os.mem.used_percent` for all nodes (coordinator/data/master) are at max: ```auto "os" : { "timestamp" : 1597067326462, "cpu" : { "percent" : 4, "load_average" : { "1m" : 0.92, "5m" : 0.94, "15m" : 0.83 } }, "mem" : { "total_in_bytes" : 66008121344, "free_in_bytes" : 755060736, "used_in_bytes" : 65253060608, "free_percent" : 1, "used_percent" : 99 }, ``` Is this normal? I am seeing this behavior in both clusters. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 10, 2020, 2:00pm UTC](https://discuss.elastic.co/t/elasticsearch-7-6-is-underperforming-6-8/244211/12 "2020-08-10T14:00:04Z")
Yes, that is normal as it includes the OS page cache. ---
**Author:** ![Steve\_Mushero](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_mushero/32/22441_2.png) [@Steve\_Mushero](https://discuss.elastic.co/u/Steve_Mushero)\ **Post date:** [August 10, 2020, 2:06pm UTC](https://discuss.elastic.co/t/elasticsearch-7-6-is-underperforming-6-8/244211/13 "2020-08-10T14:06:00Z")
Hmm, that load average of under 1 implies we are not waiting for disk very much; not what I'd expect on IO bound system with lots of parallel fsyncs() - still suggest you try async translog at some point; can change it dynamically for an hour and see if things improve; don't forget to change back. You don't have root on these VM? Without that kinda hard, but note iostat may be installed just not in your path; look for it (find / -name iostat) else 'sar' might be install and vmstat can show IO and blocked info but not disk queues - BUT you can see external disk queue in AWS; suggest take a look on the nodes in Cloud Watch. For the you posted, stats, you need to take them over time; the absolute numbers aren't that important, but how much they change say in a minute or 15 minutes is what matters to gauge if they are throttling at all now. V7 shows 8M millis of throttle vs. 84M in merging, or 10%; on 71G of size. V6 about 20%, but should get better data a few min apart. Clearly both are throttling but I have no idea if that's a lot or not. ---
**Author:** ![danksim](https://avatars.discourse-cdn.com/v4/letter/d/a88e57/32.png) [@danksim](https://discuss.elastic.co/u/danksim)\ **Post date:** [August 10, 2020, 2:28pm UTC](https://discuss.elastic.co/t/elasticsearch-7-6-is-underperforming-6-8/244211/14 "2020-08-10T14:28:23Z")
> [@Steve\_Mushero](#): > > Hmm, that load average of under 1 implies we are not waiting for disk very much; not what I'd expect on IO bound system with lots of parallel fsyncs() - still suggest you try async translog at some point; can change it dynamically for an hour and see if things improve; don't forget to change back. So if IO is good, I wonder what the problem is... > [@Steve\_Mushero](#): > > You don't have root on these VM? I do not have root. I think by default. I am using the official Elastic helm charts (see [here](https://github.com/elastic/helm-charts)). > [@Steve\_Mushero](#): > > look for it (find / -name iostat) else 'sar' might be install and vmstat can show IO and blocked info but not disk queues Luckily, it looks like I have `vmstat` (no `iostat` or `sar`): ```auto [elasticsearch@elastic-stack-data-0 ~]$ vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 5 0 5130784 394744 33181804 0 0 24 287 3 2 5 1 73 21 0 ``` Not sure what to make of it. > [@Steve\_Mushero](#): > > For the you posted, stats, you need to take them over time; the absolute numbers aren't that important, but how much they change say in a minute or 15 minutes is what matters to gauge if they are throttling at all now. V7 shows 8M millis of throttle vs. 84M in merging, or 10%; on 71G of size. V6 about 20%, but should get better data a few min apart. Clearly both are throttling but I have no idea if that's a lot or not. I was monitoring the numbers and now that number is `8940977`(from `8862331`). ---
**Author:** ![Steve\_Mushero](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_mushero/32/22441_2.png) [@Steve\_Mushero](https://discuss.elastic.co/u/Steve_Mushero)\ **Post date:** [August 10, 2020, 2:47pm UTC](https://discuss.elastic.co/t/elasticsearch-7-6-is-underperforming-6-8/244211/15 "2020-08-10T14:47:14Z")
Gee, I gotta figure out how to do nice quoting like that on this discuss platform. Overall, seems no idea; all I can think of is faster disks or set async; the latter fast & easy to try; the former costs money & time. I'm just guessing not much IO, hard to tell. And if in a container, not really sure how well it tells us about IO; kinda hazy in that sense, I'd think. For vmstat run like 'vmstat 2 5' to get 10 seconds worth of data. For ES stats, you have to get both clusters, say 1 min apart for 2-3 minutes so you can calculate the rate in throttled millis/minute to see if they are similar but not really worth the effort; just that you have some throttling but seems not huge. But your number went up 80K mills or 80 seconds which is a lot of it was only in a minute, though I don't know a lot about this stat. Sorry can't be more helpful. ---
**Author:** ![danksim](https://avatars.discourse-cdn.com/v4/letter/d/a88e57/32.png) [@danksim](https://discuss.elastic.co/u/danksim)\ **Post date:** [August 11, 2020, 6:24am UTC](https://discuss.elastic.co/t/elasticsearch-7-6-is-underperforming-6-8/244211/16 "2020-08-11T06:24:36Z")
> [@Steve\_Mushero](#): > > Overall, seems no idea; all I can think of is faster disks or set async; the latter fast & easy to try; the former costs money & time. Wow. This is incredible. Setting `index.translog.durability": "async"` greatly improved indexing throughput and got rid of all of the lag. I will have to really understand the pros and cons before permanently settling on this method but this is amazing progress. Thank you so much, Steve! ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 11, 2020, 6:56am UTC](https://discuss.elastic.co/t/elasticsearch-7-6-is-underperforming-6-8/244211/17 "2020-08-11T06:56:08Z")
> [@danksim](#): > > I will have to really understand the pros and cons before permanently settling on this method The main con is that Elasticsearch no longer guarantees that your data is written to disk before acknowledging it, so an unexpected restart may result in lost data. It's up to you to decide whether that's a big deal or not. Maybe on EC2 it's not so bad, instances tend to just vanish rather than restarting with stale data. Spinning disks really are terrible for indexing performance. Have you considered a hot/warm architecture in which you have a few small/fast "hot" nodes doing the indexing (with SSDs) and then after rollover you migrate those indices to larger/slower/cheaper "warm" nodes which only have spinning disks. There's other advice in the manual on [improving indexing speed](https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html) too; in particular larger bulks targeting fewer shards should have a similar effect to using to async durability but without weaker durability guarantees. > [@danksim](#): > > both are getting a ton of 429s That's a pretty strong indication that your cluster is underpowered. Serving 429s is something of a last resort to avoid completely blowing up the cluster and (depending on where exactly the 429s are coming from) it can be quite expensive in its own right. I recommend working towards hitting a 429 only very rarely. ---
**Author:** ![danksim](https://avatars.discourse-cdn.com/v4/letter/d/a88e57/32.png) [@danksim](https://discuss.elastic.co/u/danksim)\ **Post date:** [August 11, 2020, 7:18am UTC](https://discuss.elastic.co/t/elasticsearch-7-6-is-underperforming-6-8/244211/18 "2020-08-11T07:18:32Z")
> [@DavidTurner](#): > > The main con is that Elasticsearch no longer guarantees that your data is written to disk before acknowledging it, so an unexpected restart may result in lost data. It's up to you to decide whether that's a big deal or not. Maybe on EC2 it's not so bad, instances tend to just vanish rather than restarting with stale data. That was my understanding. Was wondering if there were any other setbacks. > [@DavidTurner](#): > > Spinning disks really are terrible for indexing performance. Have you considered a hot/warm architecture in which you have a few small/fast "hot" nodes doing the indexing (with SSDs) and then after rollover you migrate those indices to larger/slower/cheaper "warm" nodes which only have spinning disks. I have looked at this concept. I am hesitant about `SSD`s right now. `AWS` spending is pretty high as-is. But maybe I can even out the cost with lower-tier `HDD`s for the `warm` nodes. Definitely something to consider! > [@DavidTurner](#): > > There's other advice in the manual on [improving indexing speed](https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html) too; in particular larger bulks targeting fewer shards should have a similar effect to using to async durability but without weaker durability guarantees. I have looked at this and we are already doing 7/10! The 3 are `Disable swapping`, `Use faster hardware` and `Use cross-cluster replication to prevent searching from stealing resources from indexing`. > [@DavidTurner](#): > > That's a pretty strong indication that your cluster is underpowered. Serving 429s is something of a last resort to avoid completely blowing up the cluster and (depending on where exactly the 429s are coming from) it can be quite expensive in its own right. I recommend working towards hitting a 429 only very rarely. Oh my. I thought this was normal! Even well-performing 6.8.2 cluster was showing a bunch of `429`s. Hm... I will have to revisit this with my team. Maybe `SSD`s are necessary. Thank you, David! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 8, 2020, 7:18am UTC](https://discuss.elastic.co/t/elasticsearch-7-6-is-underperforming-6-8/244211/19 "2020-09-08T07:18:43Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.