I have an issue with disk space as the disk indices greater than size of index
as shown in the attached file.
can anyone help me to understand why the data much greater than actual index size as sum of storage of indicies less than 50 GB and used data is 459 GB
Lets take a look.
Assuming you are logged in as the elastic user please go to
Kibana / Dev Tools and run the following, this will list your indices in descending order of size...
GET /_cat/indices/*?v&s=pri.store.size:desc
Please do not post a screen shot please copy in the first 20-30 lines and format with the </> button from the editor
Thanks @stephenb
the result of the query is
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .ds-logs-elastic_agent.filebeat-default-2021.03.10-000004 8JP636k6T-qzK98umpFZhA 1 1 534362279 0 100.2gb 50.1gb
green open .ds-logs-elastic_agent.filebeat-default-000002 e35C7EvuQuWGOH17qHWEQA 1 1 500621698 0 99.7gb 49.8gb
green open .ds-logs-elastic_agent.filebeat-default-000001 QXBQBWwpRgOzEF0xqrcWQg 1 1 467603022 0 98.2gb 49.1gb
green open .ds-logs-elastic_agent.filebeat-default-2021.04.09-000005 -bX8mGTGRDaZgzNGmkNWtg 1 1 421159309 0 78.1gb 38.9gb
green open .ds-logs-elastic_agent.metricbeat-default-000002 R6YF1y_6QoOg94x4PAGrrw 1 1 9220875 0 8.6gb 4.3gb
green open .ds-logs-elastic_agent.metricbeat-default-000001 gBV6k5clTFygAkRHowPIpA 1 1 8664150 0 8.1gb 4gb
green open .ds-logs-elastic_agent.metricbeat-default-2021.03.10-000004 xcQdy8q_SdaFY22G01hPcg 1 1 8799618 0 7.4gb 3.7gb
green open .ds-logs-elastic_agent.metricbeat-default-000003 vnR6HABIRMSw42joM99wUg 1 1 7960187 0 6.9gb 3.4gb
green open .ds-logs-elastic_agent.metricbeat-default-2021.04.09-000005 85TnxpWOT4OpKPwNaa4bZw 1 1 5864854 0 6.1gb 3gb
green open .ds-logs-iis.error-default-000001 6Bd1eysrQUCZq_VQ7ikXxQ 1 1 13461733 0 2.6gb 1.3gb
green open packetbeat-7.10.0-2021.04.26-000020 V1HRof-LRu2zQX-EjGiWeQ 1 1 3877024 0 2.6gb 1.3gb
green open .monitoring-es-7-2021.04.22 kUlj3JzeSIeLyFydcQ08Qg 1 1 2257158 1092029 1.8gb 952.5mb
green open .monitoring-es-7-2021.04.25 t0e2E-_lRh2LV2g8nqiRsw 1 1 2694775 714400 1.8gb 936.7mb
green open .monitoring-es-7-2021.04.23 K9HF46OTQ0qAqCBNf7JINQ 1 1 2970705 872826 1.7gb 904mb
green open .monitoring-es-7-2021.04.27 oYtszhG-QO6Y_LNt5t1nqA 1 1 1547904 903922 1.4gb 736.7mb
green open ssty-index-alias gDeBd4HDRqOLyMw_hzPSew 1 1 220345 71 1.2gb 617.1mb
green open .monitoring-es-7-2021.04.26 cmTMbPMwTTKcGrT-TOPqDg 1 1 1098790 281592 979.2mb 487.2mb
green open .monitoring-es-7-2021.04.28 3OcBRVYIRleOWd0iNwI5tA 1 1 519393 290188 853.9mb 423.2mb
green open .ds-metrics-system.process-default-000002 vZS9FGwFTb-fDTB53w8Ffg 1 1 1828761 0 804.7mb 402.3mb
green open .ds-metrics-system.process-default-000001 IhKZDXzqRYKWOKpw-6JOBQ 1 1 1669125 0 743.1mb 371.5mb
green open metricbeat-7.8.0-2021.04.28-000002 lZUsJIVgQ-uJuOE8KVRueg 1 1 947695 0 694.9mb 358.1mb
green open .ds-metrics-system.process-default-2021.03.10-000004 a9EpljmnRIuwfhfMHqARVw 1 1 1599724 0 702.3mb 351.1mb
green open .ds-metrics-system.process-default-000003 wfZuecEAQ42x2Y-byXbN_w 1 1 1452452 0 627.9mb 313.9mb
green open .ds-metrics-system.process-default-2021.04.09-000005 p2cde0UcQ_2ak1eKvfzniQ 1 1 1138999 0 501.1mb 249mb
green open .ds-logs-iis.error-default-2021.03.10-000004 Vnu9Y6I8SwKOaUbhfipJzQ 1 1 2242547 0 440.6mb 220.3mb
green open apm-7.11.1-span 9zPQtPsWTBeGQDzQdCq0UA 1 1 243185 0 292.3mb 146.1mb
green open .ds-metrics-iis.website-default-2021.04.09-000005 4j-YYPv3Q3uwxPQqZ6DC_g 1 1 408394 0 289.2mb 141.3mb
green open .ds-metrics-iis.website-default-000002 HP0QcQAKTt-xr5ipjjZrhg 1 1 730001 0 275.8mb 137.9mb
green open .ds-metrics-system.network-default-2021.03.10-000004 vJXN1nzoRmqQQxsV7VdO5A 1 1 1008045 0 275mb 137.5mb
green open .ds-metrics-system.network-default-000002 5D68_r05QniTDYePkZv9cQ 1 1 943194 0 266.2mb 133.1mb
green open .ds-metrics-iis.website-default-000001 pdA9s-gfTdmVHiCpVJuBzw 1 1 672381 0 252.9mb 126.4mb
green open .ds-metrics-system.network-default-000001 siHxzINRTd6nRnzNnwaFTA 1 1 896941 0 252.6mb 126.3mb
green open .monitoring-es-7-2021.04.24 TBVxafeQQDOY7k2vX8DPMQ 1 1 433878 127148 236mb 118mb
green open .ds-metrics-iis.website-default-2021.03.10-000004 4Ood7P6DQkef3TB4Fdjyhg 1 1 613851 0 223.4mb 111.7mb
green open .ds-metrics-system.network-default-000003 9XGpNbgKQ4qYUABU-BoR4w 1 1 774040 0 214.6mb 107.3mb
green open .ds-metrics-system.network-default-2021.04.09-000005 Yq2etQvwR4qTJgN4N0PYNw 1 1 660082 0 209.9mb 104.9mb
green open .ds-metrics-iis.website-default-000003 O6xClk5HSfKlDUn6UdTnkw 1 1 559113 0 204.6mb 102.3mb
green open .ds-logs-iis.error-default-000002 SOeaPpgXQvuZaVyWLfqgdQ 1 1 958058 0 194.7mb 97.3mb
green open metricbeat-7.9.2 GQ7r2wXKROWHpxPJGXONFA 1 1 171819 0 169.7mb 84.8mb
green open .ds-metrics-iis.webserver-default-000002 yyobQ8jPTZa1zwuA-QrtmQ 1 1 244086 0 160.7mb 80.3mb
green open .ds-metrics-iis.webserver-default-000001 kPAxbkBgRqaUTsjrnNkWhA 1 1 224241 0 144.9mb 72.4mb
green open .ds-metrics-iis.webserver-default-2021.03.10-000004 YU_mCcD6SduXqSvRXfkAkA 1 1 205602 0 133.1mb 66.5mb
green open .kibana-event-log-7.10.0-000002 SVQ-lcJHSVmfl4NgH6MTJQ 1 1 1053284 0 127.6mb 63.8mb
green open .ds-logs-iis.error-default-2021.04.09-000005 sblFpCpwTkqN4IAmdAKL1g 1 1 647095 0 124.1mb 61.8mb
green open .ds-metrics-system.diskio-default-000002 KQrc2MO0SZyJyc3LqpV08g 1 1 471670 0 122.6mb 61.3mb
green open .ds-metrics-iis.webserver-default-000003 y61kzWejTFWgdha9u2jooQ 1 1 193986 0 119.6mb 59.8mb
green open .ds-metrics-system.diskio-default-000001 5lIWgf0JTymLFzMgEYVudA 1 1 448484 0 118.4mb 59.2mb
green open heartbeat-7.8.0-2021.04.19-000001 GkX1pVCpTAW8jmFwpYPA7A 1 1 120756 0 113mb 56.4mb
green open apm-7.11.1-metric J-W3Ro74TNy9C3DYraImSA 1 1 332505 0 105mb 52.4mb
green open .ds-metrics-system.diskio-default-2021.03.10-000004 U-6aIFvzT-CPpcJS32bOlw 1 1 410407 0 104.3mb 52.1mb
green open .ds-metrics-system.diskio-default-000003 gkBmJQtySUKEQpzVo_QYbA 1 1 389523 0 99.7mb 49.8mb
green open .ds-metrics-elastic_agent.filebeat-default-000002 QAgjEq0QRwaxtow6kGBDXQ 1 1 536099 0 99.3mb 49.6mb
green open .ds-metrics-system.memory-default-000002 Vb-_CRnZR0e_Z86060AtVg 1 1 246410 0 96.5mb 48.2mb
green open .ds-metrics-elastic_agent.metricbeat-default-000002 a1S3g7suQsCTZt3eyEMhRw 1 1 523413 0 96mb 48mb
green open .ds-metrics-system.memory-default-000001 r5GXO2uORzO7LOyehs6Dhw 1 1 224246 0 90mb 45mb
green open .ds-metrics-elastic_agent.filebeat-default-000003 T6P8iP9kR7aQW2sW8UUygw 1 1 478817 0 89.5mb 44.7mb
green open .ds-metrics-elastic_agent.metricbeat-default-000003 EPSgHgG7R6GLhIUP8lDa0g 1 1 473346 0 88.3mb 44.1mb
green open .ds-metrics-iis.webserver-default-2021.04.09-000005 OYNMagWvSkueFSxSBLCjRw 1 1 134685 0 88.4mb 43.7mb
green open .ds-metrics-elastic_agent.filebeat-default-000001 zG3XZ_0mRZyh2aJ74IbLqA 1 1 453700 0 83.8mb 41.9mb
green open .ds-metrics-elastic_agent.metricbeat-default-000001 D9BKc55ZSjKG0fzg9Vz0gA 1 1 453702 0 83.1mb 41.5mb
green open .ds-metrics-system.memory-default-2021.03.10-000004 V_S4HL3vQJ-yYFl7LxOAiQ 1 1 205433 0 80.5mb 40.2mb
green open apm-7.11.1-transaction 6N6JZamrQJ2YcaT2iQP_4A 1 1 84825 0 80.2mb 40mb
green open .ds-metrics-elastic_agent.filebeat-default-2021.03.10-000004 y86O1uiwSIWyW_tOSDkfXQ 1 1 424726 0 77.4mb 38.7mb
green open .ds-metrics-elastic_agent.metricbeat-default-2021.03.10-000004 FAb42q96RRSypL6-s_iTXA 1 1 423774 0 77.4mb 38.7mb
green open .ds-metrics-system.cpu-default-000002 A89f9u3LT-W9wJmEvxCITA 1 1 244379 0 75.1mb 37.5mb
green open .ds-metrics-system.cpu-default-2021.04.09-000005 RQYFMSuFRlWB9LKZ0TGFZw 1 1 130441 0 74.5mb 37.1mb
green open .ds-metrics-system.diskio-default-2021.04.09-000005 jMe29Ok5Rui5z-jHByEyjA 1 1 284766 0 73.1mb 37mb
green open .ds-metrics-system.memory-default-000003 rBL2bNU0SqGSEtqEzVt0Ag 1 1 193650 0 73.8mb 36.9mb
green open .ds-metrics-system.cpu-default-000001 jmIBxpbnTJW0lC7MKiS-VA 1 1 224245 0 68.9mb 34.4mb
green open .ds-metrics-system.cpu-default-2021.03.10-000004 zsjyN_7uT7mhFvDlg4HFcQ 1 1 205317 0 62.5mb 31.2mb
green open .ds-metrics-system.socket_summary-default-000002 WLRKtemWSjWvoeyCfIm2LQ 1 1 242189 0 58.4mb 29.2mb
green open .ds-metrics-system.cpu-default-000003 rC5wh_xkQPeM5C0mJYXVEw 1 1 190801 0 56.6mb 28.3mb
green open .ds-metrics-system.memory-default-2021.04.09-000005 pnhPhV-7QcS3bwB8dyeGwA 1 1 140152 0 56.2mb 28.2mb
green open .ds-logs-elastic_agent.filebeat-default-000003 O0FAVXwqTUCeiCQF-2nowQ 1 1 155972 0 54.7mb 27.3mb
So as you can see you have many large indices... Those top 6 indices add up to about 360GB and I am sure with all the smaller ones it all adds up.
What version are you on... I think if you click on that Filter for system indices button you will see all the .ds-* indices and sort and you will see what you see from the command above.
Those are data stream indices most likely coming from the Elastic Agent you are using to collect telemetry.
Many Thanks @stephenb
elastic search version 7.12
I knew why I could not browse .ds-* now I can see them from index management
by including hidden indices
.ds-* indices came from the data stream as I used the fleet feature and forgot to assign an index life cycle for them.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.
