# One of the nodes disk usage is almost full **URL:** \ **Category:** Elasticsearch\ **Created:** [November 15, 2022, 4:07pm UTC](https://discuss.elastic.co/t/one-of-the-nodes-disk-usage-is-almost-full/318977 "2022-11-15T16:07:40Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![catadetest](https://avatars.discourse-cdn.com/v4/letter/c/a5b964/32.png) [@catadetest](https://discuss.elastic.co/u/catadetest)\ **Post date:** [November 15, 2022, 4:07pm UTC](https://discuss.elastic.co/t/one-of-the-nodes-disk-usage-is-almost-full/318977/1 "2022-11-15T16:07:40Z")
We have 3 nodes in an elasticsearch cluster on version: 6.8.21 All of them are master and data nodes. Today, we observed that one node disk usage is almost full (88%), while the others are only 13%, 25% used. ```auto curl -s "http://10.0.6.21:9200/_cat/allocation?v&pretty" shards disk.indices disk.used disk.avail disk.total disk.percent host ip node 130 25gb 32.3gb 93.1gb 125.4gb 25 10.0.6.22 10.0.6.22 node-02a 130 9.9gb 16.5gb 108.9gb 125.4gb 13 10.0.6.23 10.0.6.23 node-03a 15 12.2gb 110.6gb 14.8gb 125.4gb 88 10.0.6.21 10.0.6.21 node-01a ``` It's funny that node01 disk is almost full, while it has the least amount of shards, only 15, compared to the other nodes which have 130 both: ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/8/584d42fed2a3bba334797900df826f146e6a516b.png) Any help would be much appreciated. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 15, 2022, 5:15pm UTC](https://discuss.elastic.co/t/one-of-the-nodes-disk-usage-is-almost-full/318977/2 "2022-11-15T17:15:43Z")
Look at the shards on that node using the [\_cat shards API](https://www.elastic.co/guide/en/elasticsearch/reference/8.5/cat-shards.html) to see what sets them apart from the rest. Do they have replica shards configured? If so, where are they located? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 16, 2022, 6:54am UTC](https://discuss.elastic.co/t/one-of-the-nodes-disk-usage-is-almost-full/318977/3 "2022-11-16T06:54:13Z")
Also upgrade your cluster. 8.5.1 is the latest version. ---
**Author:** ![catadetest](https://avatars.discourse-cdn.com/v4/letter/c/a5b964/32.png) [@catadetest](https://discuss.elastic.co/u/catadetest)\ **Post date:** [November 16, 2022, 8:52am UTC](https://discuss.elastic.co/t/one-of-the-nodes-disk-usage-is-almost-full/318977/4 "2022-11-16T08:52:28Z")
@Christian_Dahlqvist These are the shards results: ```auto index shard prirep state docs store ip node lite_pnn 2 p STARTED 64 21.8kb 10.0.6.22 node-02a lite_pnn 2 r STARTED 64 21.8kb 10.0.6.23 node-03a lite_pnn 4 p STARTED 87 32kb 10.0.6.22 node-02a lite_pnn 4 r STARTED 87 32kb 10.0.6.23 node-03a lite_pnn 1 r STARTED 83 23.8kb 10.0.6.22 node-02a lite_pnn 1 p STARTED 83 23.8kb 10.0.6.23 node-03a lite_pnn 3 p STARTED 75 43.9kb 10.0.6.22 node-02a lite_pnn 3 r STARTED 75 43.9kb 10.0.6.23 node-03a lite_pnn 0 p STARTED 75 16kb 10.0.6.22 node-02a lite_pnn 0 r STARTED 75 16kb 10.0.6.23 node-03a lite_digital_events_device 1 r STARTED 1114 1mb 10.0.6.22 node-02a lite_digital_events_device 1 p STARTED 1114 1.1mb 10.0.6.23 node-03a lite_digital_events_device 4 p STARTED 1084 1mb 10.0.6.22 node-02a lite_digital_events_device 4 r STARTED 1084 1mb 10.0.6.23 node-03a lite_digital_events_device 2 r STARTED 1179 1mb 10.0.6.22 node-02a lite_digital_events_device 2 p STARTED 1179 1mb 10.0.6.23 node-03a lite_digital_events_device 3 p STARTED 1133 1mb 10.0.6.22 node-02a lite_digital_events_device 3 r STARTED 1133 1000.3kb 10.0.6.23 node-03a lite_digital_events_device 0 p STARTED 1085 1mb 10.0.6.22 node-02a lite_digital_events_device 0 r STARTED 1085 1mb 10.0.6.23 node-03a lite_content_playlist 1 r STARTED 148 542.4kb 10.0.6.22 node-02a lite_content_playlist 1 p STARTED 148 539kb 10.0.6.23 node-03a lite_content_playlist 4 p STARTED 117 392.3kb 10.0.6.22 node-02a lite_content_playlist 4 r STARTED 117 366.6kb 10.0.6.23 node-03a lite_content_playlist 2 r STARTED 143 469.4kb 10.0.6.22 node-02a lite_content_playlist 2 p STARTED 143 462.6kb 10.0.6.23 node-03a lite_content_playlist 3 p STARTED 134 311.3kb 10.0.6.22 node-02a lite_content_playlist 3 r STARTED 134 320.8kb 10.0.6.23 node-03a lite_content_playlist 0 p STARTED 136 380.3kb 10.0.6.22 node-02a lite_content_playlist 0 r STARTED 136 380.7kb 10.0.6.23 node-03a lite_encoding_job 3 p STARTED 547333 179.3mb 10.0.6.22 node-02a lite_encoding_job 3 r STARTED 547333 179.3mb 10.0.6.23 node-03a lite_encoding_job 4 p STARTED 545352 179mb 10.0.6.22 node-02a lite_encoding_job 4 r STARTED 545352 179mb 10.0.6.23 node-03a lite_encoding_job 1 r STARTED 547771 184.3mb 10.0.6.22 node-02a lite_encoding_job 1 p STARTED 547771 184.3mb 10.0.6.23 node-03a lite_encoding_job 2 r STARTED 546427 180.6mb 10.0.6.22 node-02a lite_encoding_job 2 p STARTED 546427 180.6mb 10.0.6.23 node-03a lite_encoding_job 0 p STARTED 546681 175.9mb 10.0.6.22 node-02a lite_encoding_job 0 r STARTED 546681 175.9mb 10.0.6.23 node-03a lite_playlist 0 p STARTED 2004 193.6mb 10.0.6.21 node-01a lite_pire_audio 0 p STARTED 62308 33.6mb 10.0.6.21 node-01a lite_pire_distributed_audio 0 p STARTED 50870 54.3mb 10.0.6.23 node-03a lite_pire_distributed_document 0 p STARTED 1539775 1.7gb 10.0.6.21 node-01a lite_pire_distributed_three_dimensional_asset 0 p STARTED 3 266.6kb 10.0.6.21 node-01a lite_legacy_distribution_document 1 p STARTED 283710 133.5mb 10.0.6.22 node-02a lite_legacy_distribution_document 1 r STARTED 283710 133.5mb 10.0.6.23 node-03a lite_legacy_distribution_document 4 r STARTED 283022 134.3mb 10.0.6.22 node-02a lite_legacy_distribution_document 4 p STARTED 283022 134.2mb 10.0.6.23 node-03a lite_legacy_distribution_document 2 p STARTED 283019 133.6mb 10.0.6.22 node-02a lite_legacy_distribution_document 2 r STARTED 283019 133.6mb 10.0.6.23 node-03a lite_legacy_distribution_document 3 r STARTED 282981 133.6mb 10.0.6.22 node-02a lite_legacy_distribution_document 3 p STARTED 282981 133.5mb 10.0.6.23 node-03a lite_legacy_distribution_document 0 r STARTED 282420 132.6mb 10.0.6.22 node-02a lite_legacy_distribution_document 0 p STARTED 282420 132.6mb 10.0.6.23 node-03a lite_php_pia_content_based_playlist 2 r STARTED 378 3.6mb 10.0.6.22 node-02a lite_php_pia_content_based_playlist 2 p STARTED 378 3.6mb 10.0.6.23 node-03a lite_php_pia_content_based_playlist 4 p STARTED 365 3.8mb 10.0.6.22 node-02a lite_php_pia_content_based_playlist 4 r STARTED 365 3.8mb 10.0.6.23 node-03a lite_php_pia_content_based_playlist 3 p STARTED 384 3.4mb 10.0.6.22 node-02a lite_php_pia_content_based_playlist 3 r STARTED 384 3.4mb 10.0.6.23 node-03a lite_php_pia_content_based_playlist 1 p STARTED 357 2.7mb 10.0.6.22 node-02a lite_php_pia_content_based_playlist 1 r STARTED 357 2.7mb 10.0.6.23 node-03a lite_php_pia_content_based_playlist 0 r STARTED 336 3.1mb 10.0.6.22 node-02a lite_php_pia_content_based_playlist 0 p STARTED 336 3.1mb 10.0.6.23 node-03a lite_pire_video 0 p STARTED 2718736 5gb 10.0.6.21 node-01a lite_php_pia_video 3 p STARTED 234702 116.8mb 10.0.6.22 node-02a lite_php_pia_video 3 r STARTED 234702 116.7mb 10.0.6.23 node-03a lite_php_pia_video 4 p STARTED 233817 116.6mb 10.0.6.22 node-02a lite_php_pia_video 4 r STARTED 233817 116.6mb 10.0.6.23 node-03a lite_php_pia_video 2 p STARTED 234006 117mb 10.0.6.22 node-02a lite_php_pia_video 2 r STARTED 234006 117mb 10.0.6.23 node-03a lite_php_pia_video 1 p STARTED 235051 117.5mb 10.0.6.22 node-02a lite_php_pia_video 1 r STARTED 235051 117.5mb 10.0.6.23 node-03a lite_php_pia_video 0 r STARTED 234354 117.6mb 10.0.6.22 node-02a lite_php_pia_video 0 p STARTED 234354 117.6mb 10.0.6.23 node-03a .monitoring-kibana-6-2022.11.15 0 r STARTED 3166 892.6kb 10.0.6.22 node-02a .monitoring-kibana-6-2022.11.15 0 p STARTED 3166 892.6kb 10.0.6.21 node-01a lite_upload_task_item 2 r STARTED 504272 89mb 10.0.6.22 node-02a lite_upload_task_item 2 p STARTED 504272 87.2mb 10.0.6.23 node-03a lite_upload_task_item 4 p STARTED 502625 86.6mb 10.0.6.22 node-02a lite_upload_task_item 4 r STARTED 502625 86.6mb 10.0.6.23 node-03a lite_upload_task_item 3 p STARTED 502847 85.8mb 10.0.6.22 node-02a lite_upload_task_item 3 r STARTED 502847 85.8mb 10.0.6.23 node-03a lite_upload_task_item 1 p STARTED 503685 86.9mb 10.0.6.22 node-02a lite_upload_task_item 1 r STARTED 503685 88.8mb 10.0.6.23 node-03a lite_upload_task_item 0 r STARTED 503062 88.5mb 10.0.6.22 node-02a lite_upload_task_item 0 p STARTED 503062 88.6mb 10.0.6.23 node-03a .kibana_task_manager 0 r STARTED 2 6.8kb 10.0.6.22 node-02a .kibana_task_manager 0 p STARTED 2 12.6kb 10.0.6.21 node-01a lite_php_pia_user 2 p STARTED 1408 3.5mb 10.0.6.22 node-02a lite_php_pia_user 2 r STARTED 1408 3.5mb 10.0.6.23 node-03a lite_php_pia_user 4 p STARTED 1447 2.7mb 10.0.6.22 node-02a lite_php_pia_user 4 r STARTED 1447 2.7mb 10.0.6.23 node-03a lite_php_pia_user 1 r STARTED 1484 3.3mb 10.0.6.22 node-02a lite_php_pia_user 1 p STARTED 1484 3.3mb 10.0.6.23 node-03a lite_php_pia_user 3 r STARTED 1446 4.6mb 10.0.6.22 node-02a lite_php_pia_user 3 p STARTED 1446 4.6mb 10.0.6.23 node-03a lite_php_pia_user 0 p STARTED 1389 3.5mb 10.0.6.22 node-02a lite_php_pia_user 0 r STARTED 1389 3.5mb 10.0.6.23 node-03a lite_legacy_distribution_measurement_data_asset 1 p STARTED 0 261b 10.0.6.22 node-02a lite_legacy_distribution_measurement_data_asset 1 r STARTED 0 261b 10.0.6.23 node-03a lite_legacy_distribution_measurement_data_asset 4 r STARTED 0 261b 10.0.6.22 node-02a lite_legacy_distribution_measurement_data_asset 4 p STARTED 0 261b 10.0.6.23 node-03a lite_legacy_distribution_measurement_data_asset 2 r STARTED 0 261b 10.0.6.22 node-02a lite_legacy_distribution_measurement_data_asset 2 p STARTED 0 261b 10.0.6.23 node-03a lite_legacy_distribution_measurement_data_asset 3 r STARTED 0 261b 10.0.6.22 node-02a lite_legacy_distribution_measurement_data_asset 3 p STARTED 0 261b 10.0.6.23 node-03a lite_legacy_distribution_measurement_data_asset 0 r STARTED 0 261b 10.0.6.22 node-02a lite_legacy_distribution_measurement_data_asset 0 p STARTED 0 261b 10.0.6.23 node-03a lite_php_pia_device 1 r STARTED 1434 4.5mb 10.0.6.22 node-02a lite_php_pia_device 1 p STARTED 1434 4.5mb 10.0.6.23 node-03a lite_php_pia_device 4 p STARTED 1466 3.7mb 10.0.6.22 node-02a lite_php_pia_device 4 r STARTED 1466 3.7mb 10.0.6.23 node-03a lite_php_pia_device 2 r STARTED 1446 4.3mb 10.0.6.22 node-02a lite_php_pia_device 2 p STARTED 1446 4.3mb 10.0.6.23 node-03a lite_php_pia_device 3 p STARTED 1456 4.1mb 10.0.6.22 node-02a lite_php_pia_device 3 r STARTED 1456 4.1mb 10.0.6.23 node-03a lite_php_pia_device 0 p STARTED 1386 3.5mb 10.0.6.22 node-02a lite_php_pia_device 0 r STARTED 1386 3.5mb 10.0.6.23 node-03a lite_pire_digital_signage_playlist 0 p STARTED 1921 3.5mb 10.0.6.21 node-01a lite_composition 0 p STARTED 99 219.4kb 10.0.6.21 node-01a lite_pire_image 0 p STARTED 9410563 6.3gb 10.0.6.22 node-02a lite_distributed_image 2 p STARTED 28 82.7kb 10.0.6.22 node-02a lite_distributed_image 2 r STARTED 28 82.7kb 10.0.6.23 node-03a lite_distributed_image 4 p STARTED 22 70.7kb 10.0.6.22 node-02a lite_distributed_image 4 r STARTED 22 70.7kb 10.0.6.23 node-03a lite_distributed_image 1 r STARTED 33 90.3kb 10.0.6.22 node-02a lite_distributed_image 1 p STARTED 33 90.3kb 10.0.6.23 node-03a lite_distributed_image 3 r STARTED 22 116.8kb 10.0.6.22 node-02a lite_distributed_image 3 p STARTED 22 116.8kb 10.0.6.23 node-03a lite_distributed_image 0 p STARTED 30 176.5kb 10.0.6.22 node-02a lite_distributed_image 0 r STARTED 30 176.5kb 10.0.6.23 node-03a lite_clip 0 p STARTED 542 2.1mb 10.0.6.22 node-02a lite_php_pia_document 2 p STARTED 115052 56.1mb 10.0.6.22 node-02a lite_php_pia_document 2 r STARTED 115052 56.1mb 10.0.6.23 node-03a lite_php_pia_document 4 p STARTED 115081 56.1mb 10.0.6.22 node-02a lite_php_pia_document 4 r STARTED 115081 56.1mb 10.0.6.23 node-03a lite_php_pia_document 1 r STARTED 114733 55.8mb 10.0.6.22 node-02a lite_php_pia_document 1 p STARTED 114733 55.8mb 10.0.6.23 node-03a lite_php_pia_document 3 p STARTED 115444 55.8mb 10.0.6.22 node-02a lite_php_pia_document 3 r STARTED 115444 55.7mb 10.0.6.23 node-03a lite_php_pia_document 0 p STARTED 115561 56.5mb 10.0.6.22 node-02a lite_php_pia_document 0 r STARTED 115561 56.5mb 10.0.6.23 node-03a lite_legacy_distribution_image 3 r STARTED 1752798 694.3mb 10.0.6.22 node-02a lite_legacy_distribution_image 3 p STARTED 1752798 694mb 10.0.6.23 node-03a lite_legacy_distribution_image 4 p STARTED 1755047 703.7mb 10.0.6.22 node-02a lite_legacy_distribution_image 4 r STARTED 1755047 706mb 10.0.6.23 node-03a lite_legacy_distribution_image 2 p STARTED 1756089 702.7mb 10.0.6.22 node-02a lite_legacy_distribution_image 2 r STARTED 1756089 700.6mb 10.0.6.23 node-03a lite_legacy_distribution_image 1 r STARTED 1756266 704.3mb 10.0.6.22 node-02a lite_legacy_distribution_image 1 p STARTED 1756266 709.5mb 10.0.6.23 node-03a lite_legacy_distribution_image 0 p STARTED 1755031 701.5mb 10.0.6.22 node-02a lite_legacy_distribution_image 0 r STARTED 1755031 707.7mb 10.0.6.23 node-03a lite_distributed_video 1 r STARTED 2275 4.2mb 10.0.6.22 node-02a lite_distributed_video 1 p STARTED 2275 4.2mb 10.0.6.23 node-03a lite_distributed_video 4 p STARTED 2200 4mb 10.0.6.22 node-02a lite_distributed_video 4 r STARTED 2200 4mb 10.0.6.23 node-03a lite_distributed_video 2 r STARTED 2280 4.2mb 10.0.6.22 node-02a lite_distributed_video 2 p STARTED 2280 4.2mb 10.0.6.23 node-03a lite_distributed_video 3 r STARTED 2242 4.1mb 10.0.6.22 node-02a lite_distributed_video 3 p STARTED 2242 4.1mb 10.0.6.23 node-03a lite_distributed_video 0 p STARTED 2206 4mb 10.0.6.22 node-02a lite_distributed_video 0 r STARTED 2206 4mb 10.0.6.23 node-03a lite_motorships 3 p STARTED 2677 2.8mb 10.0.6.22 node-02a lite_motorships 3 r STARTED 2677 2.9mb 10.0.6.23 node-03a lite_motorships 4 p STARTED 2741 2.9mb 10.0.6.22 node-02a lite_motorships 4 r STARTED 2741 3.1mb 10.0.6.23 node-03a lite_motorships 2 p STARTED 2651 2.9mb 10.0.6.22 node-02a lite_motorships 2 r STARTED 2651 2.9mb 10.0.6.23 node-03a lite_motorships 1 p STARTED 2711 3.1mb 10.0.6.22 node-02a lite_motorships 1 r STARTED 2711 3.1mb 10.0.6.23 node-03a lite_motorships 0 r STARTED 2734 3.7mb 10.0.6.22 node-02a lite_motorships 0 p STARTED 2734 3.7mb 10.0.6.23 node-03a .monitoring-es-6-2022.11.16 0 r STARTED 162660 171.7mb 10.0.6.22 node-02a .monitoring-es-6-2022.11.16 0 p STARTED 162660 133.6mb 10.0.6.21 node-01a lite_lettering 0 p STARTED 11 419.7kb 10.0.6.21 node-01a lite_distributed_three_dimensional_asset 3 p STARTED 0 261b 10.0.6.22 node-02a lite_distributed_three_dimensional_asset 3 r STARTED 0 261b 10.0.6.23 node-03a lite_distributed_three_dimensional_asset 4 p STARTED 0 261b 10.0.6.22 node-02a lite_distributed_three_dimensional_asset 4 r STARTED 0 261b 10.0.6.23 node-03a lite_distributed_three_dimensional_asset 1 r STARTED 0 261b 10.0.6.22 node-02a lite_distributed_three_dimensional_asset 1 p STARTED 0 261b 10.0.6.23 node-03a lite_distributed_three_dimensional_asset 2 r STARTED 1 9.6kb 10.0.6.22 node-02a lite_distributed_three_dimensional_asset 2 p STARTED 1 9.6kb 10.0.6.23 node-03a lite_distributed_three_dimensional_asset 0 p STARTED 0 261b 10.0.6.22 node-02a lite_distributed_three_dimensional_asset 0 r STARTED 0 261b 10.0.6.23 node-03a lite_module 0 p STARTED 24 81.9kb 10.0.6.21 node-01a lite_pire_distributed_image 0 p STARTED 9208258 9.7gb 10.0.6.22 node-02a lite_php_pia_image 3 p STARTED 743804 319.7mb 10.0.6.22 node-02a lite_php_pia_image 3 r STARTED 743804 319.7mb 10.0.6.23 node-03a lite_php_pia_image 4 p STARTED 742428 321.8mb 10.0.6.22 node-02a lite_php_pia_image 4 r STARTED 742428 319.9mb 10.0.6.23 node-03a lite_php_pia_image 2 p STARTED 743085 321mb 10.0.6.22 node-02a lite_php_pia_image 2 r STARTED 743085 321mb 10.0.6.23 node-03a lite_php_pia_image 1 p STARTED 743393 321.6mb 10.0.6.22 node-02a lite_php_pia_image 1 r STARTED 743393 321.6mb 10.0.6.23 node-03a lite_php_pia_image 0 r STARTED 742105 321.8mb 10.0.6.22 node-02a lite_php_pia_image 0 p STARTED 742105 321.8mb 10.0.6.23 node-03a lite_legacy_distribution_collection 3 p STARTED 688 6.3mb 10.0.6.22 node-02a lite_legacy_distribution_collection 3 r STARTED 688 6.2mb 10.0.6.23 node-03a lite_legacy_distribution_collection 4 p STARTED 714 5mb 10.0.6.22 node-02a lite_legacy_distribution_collection 4 r STARTED 714 5mb 10.0.6.23 node-03a lite_legacy_distribution_collection 1 r STARTED 738 6.2mb 10.0.6.22 node-02a lite_legacy_distribution_collection 1 p STARTED 738 6.2mb 10.0.6.23 node-03a lite_legacy_distribution_collection 2 p STARTED 718 5.6mb 10.0.6.22 node-02a lite_legacy_distribution_collection 2 r STARTED 718 5.6mb 10.0.6.23 node-03a lite_legacy_distribution_collection 0 p STARTED 722 6.1mb 10.0.6.22 node-02a lite_legacy_distribution_collection 0 r STARTED 722 6.1mb 10.0.6.23 node-03a lite_distributed_measurement_data_asset 3 p STARTED 0 261b 10.0.6.22 node-02a lite_distributed_measurement_data_asset 3 r STARTED 0 261b 10.0.6.23 node-03a lite_distributed_measurement_data_asset 4 p STARTED 0 261b 10.0.6.22 node-02a lite_distributed_measurement_data_asset 4 r STARTED 0 261b 10.0.6.23 node-03a lite_distributed_measurement_data_asset 1 p STARTED 0 261b 10.0.6.22 node-02a lite_distributed_measurement_data_asset 1 r STARTED 0 261b 10.0.6.23 node-03a lite_distributed_measurement_data_asset 2 p STARTED 0 261b 10.0.6.22 node-02a lite_distributed_measurement_data_asset 2 r STARTED 0 261b 10.0.6.23 node-03a lite_distributed_measurement_data_asset 0 r STARTED 0 261b 10.0.6.22 node-02a lite_distributed_measurement_data_asset 0 p STARTED 0 261b 10.0.6.23 node-03a lite_legacy_distribution_audio 3 p STARTED 9521 3.9mb 10.0.6.22 node-02a lite_legacy_distribution_audio 3 r STARTED 9521 3.9mb 10.0.6.23 node-03a lite_legacy_distribution_audio 4 p STARTED 9539 3.9mb 10.0.6.22 node-02a lite_legacy_distribution_audio 4 r STARTED 9539 3.9mb 10.0.6.23 node-03a lite_legacy_distribution_audio 1 p STARTED 9488 3.9mb 10.0.6.22 node-02a lite_legacy_distribution_audio 1 r STARTED 9488 3.9mb 10.0.6.23 node-03a lite_legacy_distribution_audio 2 r STARTED 9450 3.9mb 10.0.6.22 node-02a lite_legacy_distribution_audio 2 p STARTED 9450 3.9mb 10.0.6.23 node-03a lite_legacy_distribution_audio 0 r STARTED 9559 3.9mb 10.0.6.22 node-02a lite_legacy_distribution_audio 0 p STARTED 9559 3.9mb 10.0.6.23 node-03a lite_device 0 p STARTED 7191 36.8mb 10.0.6.23 node-03a .kibana_1 0 r STARTED 4 19.9kb 10.0.6.23 node-03a .kibana_1 0 p STARTED 4 19.9kb 10.0.6.21 node-01a lite_legacy_distribution_video 1 r STARTED 471070 189.5mb 10.0.6.22 node-02a lite_legacy_distribution_video 1 p STARTED 471070 189.5mb 10.0.6.23 node-03a lite_legacy_distribution_video 4 p STARTED 469744 193.8mb 10.0.6.22 node-02a lite_legacy_distribution_video 4 r STARTED 469744 247.8mb 10.0.6.23 node-03a lite_legacy_distribution_video 3 p STARTED 471479 190.5mb 10.0.6.22 node-02a lite_legacy_distribution_video 3 r STARTED 471479 190.4mb 10.0.6.23 node-03a lite_legacy_distribution_video 2 p STARTED 470580 193.3mb 10.0.6.22 node-02a lite_legacy_distribution_video 2 r STARTED 470580 193.3mb 10.0.6.23 node-03a lite_legacy_distribution_video 0 p STARTED 471813 196.1mb 10.0.6.22 node-02a lite_legacy_distribution_video 0 r STARTED 471813 196.1mb 10.0.6.23 node-03a .monitoring-kibana-6-2022.11.16 0 r STARTED 3002 819.8kb 10.0.6.23 node-03a .monitoring-kibana-6-2022.11.16 0 p STARTED 3002 819.8kb 10.0.6.21 node-01a lite_distributed_audio 1 r STARTED 0 261b 10.0.6.22 node-02a lite_distributed_audio 1 p STARTED 0 261b 10.0.6.23 node-03a lite_distributed_audio 4 r STARTED 0 261b 10.0.6.22 node-02a lite_distributed_audio 4 p STARTED 0 261b 10.0.6.23 node-03a lite_distributed_audio 3 r STARTED 0 261b 10.0.6.22 node-02a lite_distributed_audio 3 p STARTED 0 261b 10.0.6.23 node-03a lite_distributed_audio 2 p STARTED 0 261b 10.0.6.22 node-02a lite_distributed_audio 2 r STARTED 0 261b 10.0.6.23 node-03a lite_distributed_audio 0 p STARTED 0 261b 10.0.6.22 node-02a lite_distributed_audio 0 r STARTED 0 261b 10.0.6.23 node-03a lite_pire_distributed_measurement_data_asset 0 p STARTED 0 261b 10.0.6.21 node-01a lite_distributed_document 3 r STARTED 15 40.6kb 10.0.6.22 node-02a lite_distributed_document 3 p STARTED 15 40.6kb 10.0.6.23 node-03a lite_distributed_document 4 p STARTED 15 45kb 10.0.6.22 node-02a lite_distributed_document 4 r STARTED 15 45kb 10.0.6.23 node-03a lite_distributed_document 1 r STARTED 13 39.8kb 10.0.6.22 node-02a lite_distributed_document 1 p STARTED 13 39.8kb 10.0.6.23 node-03a lite_distributed_document 2 p STARTED 12 31.5kb 10.0.6.22 node-02a lite_distributed_document 2 r STARTED 12 31.5kb 10.0.6.23 node-03a lite_distributed_document 0 p STARTED 20 48.7kb 10.0.6.22 node-02a lite_distributed_document 0 r STARTED 20 48.7kb 10.0.6.23 node-03a lite_php_pia_audio 2 r STARTED 5161 2.4mb 10.0.6.22 node-02a lite_php_pia_audio 2 p STARTED 5161 2.4mb 10.0.6.23 node-03a lite_php_pia_audio 4 p STARTED 5168 2.4mb 10.0.6.22 node-02a lite_php_pia_audio 4 r STARTED 5168 2.3mb 10.0.6.23 node-03a lite_php_pia_audio 3 p STARTED 5068 2.3mb 10.0.6.22 node-02a lite_php_pia_audio 3 r STARTED 5068 2.4mb 10.0.6.23 node-03a lite_php_pia_audio 1 p STARTED 5219 2.5mb 10.0.6.22 node-02a lite_php_pia_audio 1 r STARTED 5219 2.5mb 10.0.6.23 node-03a lite_php_pia_audio 0 r STARTED 5112 2.3mb 10.0.6.22 node-02a lite_php_pia_audio 0 p STARTED 5112 2.3mb 10.0.6.23 node-03a lite_pire_document 0 p STARTED 1568884 992mb 10.0.6.23 node-03a .monitoring-es-6-2022.11.15 0 r STARTED 165677 131.5mb 10.0.6.23 node-03a .monitoring-es-6-2022.11.15 0 p STARTED 165677 131.7mb 10.0.6.21 node-01a lite_pire_distributed_video 0 p STARTED 2640154 5.2gb 10.0.6.21 node-01a ``` It seems some of the shards have replicas, but some other are just primary. Some of them are only present on node-01a, some others on another nodes... ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 16, 2022, 8:59am UTC](https://discuss.elastic.co/t/one-of-the-nodes-disk-usage-is-almost-full/318977/5 "2022-11-16T08:59:03Z")
I do not understand the reasoning behind your sharding. You have lots of very small indices having multiple primary and replica shards while the largest indices has a single primary shard and no replica. Am not surprised that leads to an uneven distribution given that Elasticsearch primarily spreads data based on shard count. You can try to manually reroute some of the indices or perhaps split them up using the split index API to get a more even size. ---
**Author:** ![catadetest](https://avatars.discourse-cdn.com/v4/letter/c/a5b964/32.png) [@catadetest](https://discuss.elastic.co/u/catadetest)\ **Post date:** [November 16, 2022, 10:19am UTC](https://discuss.elastic.co/t/one-of-the-nodes-disk-usage-is-almost-full/318977/6 "2022-11-16T10:19:20Z")
Well this makes totally sense now Christian. I'll take a look and see which option suits us well, and I'll post the results here soon. Thank you so much for the info @Christian_Dahlqvist ---
**Author:** ![catadetest](https://avatars.discourse-cdn.com/v4/letter/c/a5b964/32.png) [@catadetest](https://discuss.elastic.co/u/catadetest)\ **Post date:** [November 18, 2022, 11:27am UTC](https://discuss.elastic.co/t/one-of-the-nodes-disk-usage-is-almost-full/318977/7 "2022-11-18T11:27:07Z")
Well, thanks Christian for pointing me to the shard list. Some of them don't have a replica at all, which is bad for redundancy reasons. I'm about to set replica for those shards. However, it wasn't the reason for the disk filling on that particular node, but actually, elasticsearch crashed few times, and every time it did, it dumped a _hprof_ file. There were 6 files in total, each of 16GB in size which consumed the disk. First, it seems the garbage collector takes a lot of time to release the objects: ```auto [2022-11-15T15:01:31,978][WARN][o.e.m.j.JvmGcMonitorService] [node-01a] [gc][5337794] overhead, spent [4.6s] collecting in the last [4.8s] [2022-11-15T15:01:35,213][WARN][o.e.m.j.JvmGcMonitorService] [node-01a] [gc][5337795] overhead, spent [3s] collecting in the last [3.2s] [2022-11-15T15:01:39,613][WARN][o.e.m.j.JvmGcMonitorService] [node-01a] [gc][5337796] overhead, spent [4.2s] collecting in the last [4.3s] [2022-11-15T15:01:41,556][WARN][o.e.m.o.O.Swap] [node-01a] cannot compute used swap when total swap is 0 and free swap is 0 [2022-11-15T15:01:41,570][WARN][o.e.m.j.JvmGcMonitorService] [node-01a] [gc][5337797] overhead, spent [1.7s] collecting in the last [1.8s] [2022-11-15T15:01:43,525][WARN][o.e.m.j.JvmGcMonitorService] [node-01a] [gc][5337798] overhead, spent [1.8s] collecting in the last [2s] [2022-11-15T15:01:45,292][WARN][o.e.m.j.JvmGcMonitorService] [node-01a] [gc][5337799] overhead, spent [1.6s] collecting in the last [1.7s] [2022-11-15T15:01:47,248][WARN][o.e.m.j.JvmGcMonitorService] [node-01a] [gc][5337800] overhead, spent [1.9s] collecting in the last [1.9s] [2022-11-15T15:01:50,908][WARN][o.e.m.j.JvmGcMonitorService] [node-01a] [gc][5337801] overhead, spent [3.6s] collecting in the last [1.6s] ``` Then, it eventually crashed with OOM: ```auto [2022-11-15T15:07:53,367][ERROR][o.e.ExceptionsHelper] [node-01a] fatal error [2022-11-15T15:07:53,367][ERROR][o.e.ExceptionsHelper] [node-01a] fatal error at org.elasticsearch.ExceptionsHelper.lambda$maybeDieOnAnotherThread$2(ExceptionsHelper.java:307) at java.base/java.util.Optional.ifPresent(Optional.java:183) at org.elasticsearch.ExceptionsHelper.maybeDieOnAnotherThread(ExceptionsHelper.java:297) at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.exceptionCaught(Netty4MessageChannelHandler.java:61) at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285) at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:856) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:364) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-15T15:07:53,373][ERROR][o.e.ExceptionsHelper] [node-01a] fatal error at org.elasticsearch.ExceptionsHelper.lambda$maybeDieOnAnotherThread$2(ExceptionsHelper.java:307) at java.base/java.util.Optional.ifPresent(Optional.java:183) at org.elasticsearch.ExceptionsHelper.maybeDieOnAnotherThread(ExceptionsHelper.java:297) at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.exceptionCaught(Netty4MessageChannelHandler.java:61) at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285) at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:856) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:364) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) at java.base/java.lang.Thread.run(Thread.java:829) [2022-11-15T15:07:53,374][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-01a] fatal error in thread [Thread-380574], exiting java.lang.OutOfMemoryError: Java heap space [2022-11-15T15:07:53,374][WARN][o.e.t.TcpTransport] [node-01a] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.0.6.21:45204, remoteAddress=10.0.6.22/10.0.6.22:9300}], closing connection java.lang.Exception: java.lang.OutOfMemoryError: Java heap space at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.exceptionCaught(Netty4MessageChannelHandler.java:66) [transport-netty4-client-6.8.21.jar:6.8.21] at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:856) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:364) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [netty-codec-4.1.32.Final.jar:4.1.32.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final] at java.lang.Thread.run(Thread.java:829) [?:?] Caused by: java.lang.OutOfMemoryError: Java heap space [2022-11-15T15:07:53,379][INFO][o.e.d.z.ZenDiscovery] [node-01a] master_left [{audi-mmds-es02a}{TKaKYCZ0SJydmXtDlqbOvQ}{puMzEDL7RmyIRu61kfWXmA}{10.0.6.22}{10.0.6.22:9300}{client=true, ml.machine_memory=33677197312, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [transport disconnected] [2022-11-15T15:07:53,369][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-01a] fatal error in thread [Thread-380573], exiting java.lang.OutOfMemoryError: Java heap space at org.apache.lucene.util.CharsRef.toString(CharsRef.java:122) ~[lucene-core-7.7.3.jar:7.7.3 1a0d2a901dfec93676b0fe8be425101ceb754b85 - noble - 2020-04-21 10:31:55] at org.elasticsearch.common.io.stream.StreamInput.readString(StreamInput.java:405) ~[elasticsearch-6.8.21.jar:6.8.21] at org.elasticsearch.common.io.stream.StreamInput.readGenericValue(StreamInput.java:530) ~[elasticsearch-6.8.21.jar:6.8.21] at org.elasticsearch.common.settings.Settings.readSettingsFromStream(Settings.java:566) ~[elasticsearch-6.8.21.jar:6.8.21] at org.elasticsearch.cluster.metadata.IndexMetaData.readFrom(IndexMetaData.java:777) ~[elasticsearch-6.8.21.jar:6.8.21] at org.elasticsearch.cluster.metadata.MetaData.readFrom(MetaData.java:834) ~[elasticsearch-6.8.21.jar:6.8.21] at org.elasticsearch.cluster.ClusterState.readFrom(ClusterState.java:754) ~[elasticsearch-6.8.21.jar:6.8.21] at org.elasticsearch.action.admin.cluster.state.ClusterStateResponse.lambda$readFrom$0(ClusterStateResponse.java:96) ~[elasticsearch-6.8.21.jar:6.8.21] at org.elasticsearch.action.admin.cluster.state.ClusterStateResponse$$Lambda$3465/0x0000000840c01840.read(Unknown Source) ~[?:?] at org.elasticsearch.common.io.stream.StreamInput.readOptionalWriteable(StreamInput.java:777) ~[elasticsearch-6.8.21.jar:6.8.21] at org.elasticsearch.action.admin.cluster.state.ClusterStateResponse.readFrom(ClusterStateResponse.java:96) ~[elasticsearch-6.8.21.jar:6.8.21] at org.elasticsearch.action.support.master.TransportMasterNodeAction.read(TransportMasterNodeAction.java:114) ~[elasticsearch-6.8.21.jar:6.8.21] at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$$Lambda$3451/0x0000000840bf5840.read(Unknown Source) ~[?:?] at org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:69) ~[elasticsearch-6.8.21.jar:6.8.21] at org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:36) ~[elasticsearch-6.8.21.jar:6.8.21] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1107) ~[elasticsearch-6.8.21.jar:6.8.21] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1094) ~[elasticsearch-6.8.21.jar:6.8.21] at org.elasticsearch.transport.TcpTransport.handleResponse(TcpTransport.java:970) ~[elasticsearch-6.8.21.jar:6.8.21] at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:952) ~[elasticsearch-6.8.21.jar:6.8.21] at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:763) ~[elasticsearch-6.8.21.jar:6.8.21] at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:53) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) ~[?:?] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?] ``` This is happening every 2-3 months I would say, and I have no idea why 🙂 ---
**Author:** ![catadetest](https://avatars.discourse-cdn.com/v4/letter/c/a5b964/32.png) [@catadetest](https://discuss.elastic.co/u/catadetest)\ **Post date:** [November 29, 2022, 9:43am UTC](https://discuss.elastic.co/t/one-of-the-nodes-disk-usage-is-almost-full/318977/8 "2022-11-29T09:43:06Z")
I've created replica for all the missing ones. Also, I've found the index that causes elasticsearch heap to get full, causing node crash and dumping the memory file. All the nodes have 8GB set for the heap. When searching inside a specific index, _dis\_video_, heap gets full and then the node crashes: `curl -XGET 'http://localhost:9200/dis_video/dis_video/_search?search_type=query_then_fetch' -H 'Content-Type: application/json' -d '{"query":{"bool":{"must":[{"bool":{"must":[{"query_string":{"query":"*testtesttest*"}}]}},{"bool":{"must_not":[{"term":{"status":{"value":-1,"boost":1}}}]}}]}},"from":0,"sort":{"id":{"order":"desc"},"_score":{"order":"desc"}},"size":0}'` This index, it's only 2.3MB in size: ```auto health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open dis_video 710vaYGFGEyKZqpqVkjVAs 5 0 1651 2 2.3mb 2.3mb ``` How is it possible when searching inside a 2.3mb index, causes the heap which is 8GB, to get full? ---
**Author:** ![catadetest](https://avatars.discourse-cdn.com/v4/letter/c/a5b964/32.png) [@catadetest](https://discuss.elastic.co/u/catadetest)\ **Post date:** [November 29, 2022, 2:44pm UTC](https://discuss.elastic.co/t/one-of-the-nodes-disk-usage-is-almost-full/318977/9 "2022-11-29T14:44:39Z")
So, I've did more research and it seems the culprit is a query: `curl -XGET 'http://localhost:9200/dis_video/dis_video/_search?search_type=query_then_fetch' -H 'Content-Type: application/json' -d '{"query":{"bool":{"must":[{"bool":{"must":[{"query_string":{"query":"*testtttttttttttttttttt*"}}]}},{"bool":{"must_not":[{"term":{"status":{"value":-1,"boost":1}}}]}}]}},"from":0,"sort":{"id":{"order":"desc"},"_score":{"order":"desc"}},"size":0}'` Upon execution, I did follow the java heap memory in real time, which very soon reached 100% causing elasticsearch to crash and no output for the query. ```auto heap.current heap.percent heap.max 7.9gb 100 7.9gb ``` I wasn't sure if this is because of the query, or the current elasticsearch version is having issues. So I did upgrade elasticsearch from 6.8.21 to 7.17.7 and tried the same query, which was not executed actually, throwing this error: ```auto {"error":{"root_cause":[{"type":"query_shard_exception","reason":"failed to create query: field expansion for [*] matches too many fields, limit: 1024, got: 58043","index_uuid":"iBH67VQTScWSCyDiFAU_7w","index":"dis_video"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"dis_video","node":"wLL8Vv76Sy-Fr3fJpGJDdg","reason":{"type":"query_shard_exception","reason":"failed to create query: field expansion for [*] matches too many fields, limit: 1024, got: 58043","index_uuid":"iBH67VQTScWSCyDiFAU_7w","index":"dis_video","caused_by":{"type":"illegal_argument_exception","reason":"field expansion for [*] matches too many fields, limit: 1024, got: 58043"}}}]},"status":400} ``` Seems like this newer version of Elasticsearch(7.17.7) knows the query is dangerous. Googling for it, the very first result is the following: "_The default value is 1000. The limit is in place to prevent mappings and searches from becoming too large. Higher values can lead to performance degradations and memory issues, especially in clusters with a high load or few resources_" If someone has the same issue, this research may be helpful. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 27, 2022, 2:44pm UTC](https://discuss.elastic.co/t/one-of-the-nodes-disk-usage-is-almost-full/318977/10 "2022-12-27T14:44:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.