Yesterday I set up my cluster from one node and noticed that several irrelevant indexes and ~5 million documents are displayed in Kibana monitoring. I removed the excess indexes, but the number of documents has not decreased. I tried to delete all documents with
I already deleted the indexes I wanted. But after deleting the number of documents has not changed. Its over 5 millions.
I thought an index is a file that contains logs with a certain label. To increase the processing speed, the index is divided into shards, which are distributed among the nodes of the cluster and can be processed in parallel.
It seemed to me that after the removal of the indexes winsrv-2017.01.27 - winsrv-2019.02.27 the number of documents should have decreased, but it has not changed. The amount of free space also remained the same.
I don't know where the page you are looking at gets its numbers from, nor how often it refreshes them, but perhaps they are stale or perhaps it is hiding some of the indices for some reason.
If you execute GET /_cat/indices?v in the developer console, what does it return?
Command 'GET' not found, but can be installed with:
apt install libwww-perl
I installed it, but command doesnt return anything
root@elk:~# GET /_cat/indices?v
root@elk:~#
May curl replace GET in this scenario?
root@elk:~# curl http://localhost:9200/_aliases
{".kibana_1":{"aliases":{".kibana":{}}},".monitoring-es-6-2019.02.26":{"aliases":{}},".monitoring-es-6-2019.02.27":{"aliases":{}},".monitoring-es-6-2019.02.28":{"aliases":{}},".monitoring-es-6-2019.03.01":{"aliases":{}},".monitoring-kibana-6-2019.02.26":{"aliases":{}},".monitoring-kibana-6-2019.02.27":{"aliases":{}},".monitoring-kibana-6-2019.02.28":{"aliases":{}},".monitoring-kibana-6-2019.03.01":{"aliases":{}},"kibana_sample_data_logs":{"aliases":{}},"winsrv-2019.02.28":{"aliases":{}},"winsrv-2019.03.01":{"aliases":{}}}
root@elk:~# curl 'localhost:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .monitoring-kibana-6-2019.03.01 C1Um4omRSKmI15KjHKSbUg 1 0 9537 0 5.1mb 5.1mb
green open .monitoring-es-6-2019.02.28 g9347X4KRGa0g87PlwghRQ 1 0 3531448 27192 1.5gb 1.5gb
green open .monitoring-kibana-6-2019.02.28 -q4eivhsQDOxdeOv4nT93A 1 0 10388 0 3mb 3mb
green open .monitoring-kibana-6-2019.02.26 7fqpLp5wRfqvVHPDat-utA 1 0 11689 0 2.6mb 2.6mb
green open .monitoring-kibana-6-2019.02.27 6yRTvXYESkS53FQ3_5srrg 1 0 14961 0 3.4mb 3.4mb
green open .monitoring-es-6-2019.02.26 8-wy5QDXR1iXW-jHnH2e8w 1 0 416946 13902 203.3mb 203.3mb
green open .kibana_1 2M9hIrowRU-Usn9l9y1Z_w 1 0 17 2 90.4kb 90.4kb
green open .monitoring-es-6-2019.03.01 HMvq-7pdQ06VrEXxye-f5A 1 0 59649 100 42.8mb 42.8mb
green open winsrv-2019.03.01 NE5l5-KKSpqfOnXtrjK97Q 5 0 121 0 1mb 1mb
green open winsrv-2019.02.28 hbCgWABGRHymlGE3WaBtJA 5 0 104396 0 127.5mb 127.5mb
green open .monitoring-es-6-2019.02.27 l4ueGl2HS3y2CEP7awCnfw 1 0 1151331 34196 503.2mb 503.2mb
green open kibana_sample_data_logs EBRbGki-TC-Z1LLc-CwtOQ 1 0 14005 0 11.4mb 11.4mb
When yesterday i deleted indices they stayed in console for some time, but they was greyed out.
Ok, that adds up to ~5M documents and ~2.4GB mostly in .monitoring-es-6-2019.02.28 and .monitoring-es-6-2019.02.27 so the numbers in the UI look right to me. These count as "system indices" so I think if you switch that on in Kibana you will see details for these there too.
I note that you are using daily indices for your winsrv data and have left it with the default number_of_shards (i.e. 5, the pri column). These indices look quite small (so far) so you might prefer to use a longer time range and fewer shards to get each shard into the 20-40GB range and avoid the pitfalls of having too many shards:
Maybe also consider monthly indices: winsrv-%{+YYYY.MM}. Depends on how much traffic you're anticipating, but yesterday's index was 127MB so if it carries on like that a whole month will only be ~4GB which is still pretty small.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.