# Elasticsearch Cluster Disk Write Performance

**URL:** <https://discuss.elastic.co/t/elasticsearch-cluster-disk-write-performance/350711>\
**Category:** Elasticsearch\
**Created:** [January 10, 2024, 8:25am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-disk-write-performance/350711 "2024-01-10T08:25:34Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![sheng855174](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sheng855174/32/130662_2.png) [@sheng855174](https://discuss.elastic.co/u/sheng855174)\
**Post date:** [January 10, 2024, 8:25am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-disk-write-performance/350711/1 "2024-01-10T08:25:34Z")

</div>

Hello everyone,

I have an ELK cluster and encountered performance problems, which **caused most data to be written 10 minutes slower than the actual time**. This problem occurs occasionally.

I want to know the cause of this problem and solve it.

**The following is information related to my ELK ...**

ELK node1~node9, there are nine units in total, each with the same specifications, 36 core+256 GB RAM+disk SSD 8TB.

According to my observation, there are no performance issues with CPU or RAM.

elasticsearch.yml :

```auto
node.name: node7
node.master: true
node.data: true
cluster.initial_master_nodes: ["node1", "node2", "node3"]
network.host: 0.0.0.0
discovery.seed_hosts: ["10.0.0.1", "10.0.0.2", "10.0.0.3", "10.0.0.4", "10.0.0.5","10.0.0.6","10.0.0.7","10.0.0.8","10.0.0.9"]
xpack.security.enabled: true
xpack.license.self_generated.type: basic
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/test.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/test.p12
path.data: /mnt/es1, /mnt/es2, /mnt/es3, /mnt/es4, /mnt/es5, /mnt/es6, /mnt/es7
path.logs: /var/log/elasticsearch
indices.breaker.total.use_real_memory: false
indices.breaker.total.limit: 80%
indices.fielddata.cache.size: 30%
indices.breaker.fielddata.limit: 40%
indices.breaker.request.limit: 60%
thread_pool:
  write:
    queue_size: 4000

```

  

In node7 Performance related information :

```auto
avg-cpu: %user %nice %system %iowait %steal %idle
          15.72 0.00 0.33 36.22 0.00 47.73

Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz aqu-sz %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 7.00 47.50 0.00 0.00 3.00 6.79 0.00 0.00 0.00 0.00 0.00 0.00 0.02 2.80
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 95.00 2821.00 0.00 0.00 18.85 29.69 0.00 0.00 0.00 0.00 0.00 0.00 1.79 104.00
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 4.00 57.00 0.00 0.00 40.75 14.25 0.00 0.00 0.00 0.00 0.00 0.00 0.16 8.80
sdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdh 0.00 0.00 0.00 0.00 0.00 0.00 30.00 625.00 0.00 0.00 26.03 20.83 0.00 0.00 0.00 0.00 0.00 0.00 0.78 20.80

```

Whenever a problem occurs, the writing performance of a certain hard disk (ssd) in node7 will be particularly high.

**How to confirm the cause of the problem? Is it because of hard drive damage or performance bottleneck?**

---

<div class="post-metadata">

**Author:** ![yago82](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yago82/32/97755_2.png) [@yago82](https://discuss.elastic.co/u/yago82)\
**Post date:** [January 10, 2024, 9:22am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-disk-write-performance/350711/2 "2024-01-10T09:22:16Z")

</div>

> [@sheng855174](#):
>
> I have an ELK cluster and encountered performance problems, which **caused most data to be written 10 minutes slower than the actual time**. This problem occurs occasionally.
> 
> I want to know the cause of this problem and solve it.
> 
> **The following is information related to my ELK ...**
> 
> ELK node1~node9, there are nine units in total, each with the same specifications, 36 core+256 GB RAM+disk SSD 8TB.
> 
> According to my observation, there are no performance issues with CPU or RAM.
> 
> elasticsearch.yml :
> 
> ```auto
> node.name: node7
> node.master: true
> node.data: true
> cluster.initial_master_nodes: ["node1", "node2", "node3"]
> network.host: 0.0.0.0
> discovery.seed_hosts: ["10.0.0.1", "10.0.0.2", "10.0.0.3", "10.0.0.4", "10.0.0.5","10.0.0.6","10.0.0.7","10.0.0.8","10.0.0.9"]
> xpack.security.enabled: true
> xpack.license.self_generated.type: basic
> xpack.security.transport.ssl.enabled: true
> xpack.security.transport.ssl.verification_mode: certificate
> xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/test.p12
> xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/test.p12
> path.data: /mnt/es1, /mnt/es2, /mnt/es3, /mnt/es4, /mnt/es5, /mnt/es6, /mnt/es7
> path.logs: /var/log/elasticsearch
> indices.breaker.total.use_real_memory: false
> indices.breaker.total.limit: 80%
> indices.fielddata.cache.size: 30%
> indices.breaker.fielddata.limit: 40%
> indices.breaker.request.limit: 60%
> thread_pool:
> write:
> queue_size: 4000
> 
> ```
> 
> In node7 Performance related information :
> 
> ```auto
> avg-cpu: %user %nice %system %iowait %steal %idle
> 15.72 0.00 0.33 36.22 0.00 47.73
> 
> Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz aqu-sz %util
> sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> sdb 0.00 0.00 0.00 0.00 0.00 0.00 7.00 47.50 0.00 0.00 3.00 6.79 0.00 0.00 0.00 0.00 0.00 0.00 0.02 2.80
> sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> sdd 0.00 0.00 0.00 0.00 0.00 0.00 95.00 2821.00 0.00 0.00 18.85 29.69 0.00 0.00 0.00 0.00 0.00 0.00 1.79 104.00
> sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> sdf 0.00 0.00 0.00 0.00 0.00 0.00 4.00 57.00 0.00 0.00 40.75 14.25 0.00 0.00 0.00 0.00 0.00 0.00 0.16 8.80
> sdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> sdh 0.00 0.00 0.00 0.00 0.00 0.00 30.00 625.00 0.00 0.00 26.03 20.83 0.00 0.00 0.00 0.00 0.00 0.00 0.78 20.80
> 
> ```
> 
> Whenever a problem occurs, the writing performance of a certain hard disk (ssd) in node7 will be particularly high.
> 
> **How to confirm the cause of the problem? Is it because of hard drive damage or performance bottleneck?**

Hi,

You can check the Elasticsearch slow logs: Slow logs can help you identify any slow queries or indexing operations that might be causing high disk I/O. You can enable slow logs by updating your index settings

> **[Slow Log | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-slowlog.html)**

Regards

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 7, 2024, 9:22am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-disk-write-performance/350711/3 "2024-02-07T09:22:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.