Questions about documents and indices

KOTOXJle6 · March 1, 2019, 8:23am

Hello!

Yesterday I set up my cluster from one node and noticed that several irrelevant indexes and ~5 million documents are displayed in Kibana monitoring. I removed the excess indexes, but the number of documents has not decreased. I tried to delete all documents with

curl -H'Content-Type: application/json' -XPOST 'localhost:9200/twitter/tweet/_delete_by_query?conflicts=proceed&pretty' -d'`

{
"query": {
"match_all": {}
"index" : "twitter"
}
}'

But got error

  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index",
        "resource.type" : "index_or_alias",
        "resource.id" : "twitter",
        "index_uuid" : "_na_",
        "index" : "twitter"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index",
    "resource.type" : "index_or_alias",
    "resource.id" : "twitter",
    "index_uuid" : "_na_",
    "index" : "twitter"
  },
  "status" : 404
}

How i can delete documents that related to the deleted indexes?

Kibana's elasticsearch monitoring page shows that i have 12 indices and 20 shards. But i can see names of only 3 indices.

Is that normal? How does this happens?

Im very new to ELK, so sorry if my questions are pretty weird or stupid.

DavidTurner · March 1, 2019, 9:07am

KOTOXJle6:

"root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index",
        "resource.type" : "index_or_alias",
        "resource.id" : "twitter",
        "index_uuid" : "_na_",
        "index" : "twitter"
      }

You're trying to delete documents from an index called twitter but you don't seem to have such an index.

Also I think you are trying to delete the whole index, not just its contents, so perhaps you mean:

DELETE /<index-name>

e.g.

DELETE /kibana_sample_data_logs

assuming that's one of the indices you want to delete?

KOTOXJle6 · March 1, 2019, 10:04am

I already deleted the indexes I wanted. But after deleting the number of documents has not changed. Its over 5 millions.

I thought an index is a file that contains logs with a certain label. To increase the processing speed, the index is divided into shards, which are distributed among the nodes of the cluster and can be processed in parallel.
It seemed to me that after the removal of the indexes winsrv-2017.01.27 - winsrv-2019.02.27 the number of documents should have decreased, but it has not changed. The amount of free space also remained the same.

DavidTurner · March 1, 2019, 10:12am

I don't know where the page you are looking at gets its numbers from, nor how often it refreshes them, but perhaps they are stale or perhaps it is hiding some of the indices for some reason.

If you execute GET /_cat/indices?v in the developer console, what does it return?

KOTOXJle6 · March 1, 2019, 10:31am

It returned

Command 'GET' not found, but can be installed with:

apt install libwww-perl

I installed it, but command doesnt return anything

root@elk:~# GET /_cat/indices?v
root@elk:~#

May curl replace GET in this scenario?

root@elk:~# curl http://localhost:9200/_aliases
{".kibana_1":{"aliases":{".kibana":{}}},".monitoring-es-6-2019.02.26":{"aliases":{}},".monitoring-es-6-2019.02.27":{"aliases":{}},".monitoring-es-6-2019.02.28":{"aliases":{}},".monitoring-es-6-2019.03.01":{"aliases":{}},".monitoring-kibana-6-2019.02.26":{"aliases":{}},".monitoring-kibana-6-2019.02.27":{"aliases":{}},".monitoring-kibana-6-2019.02.28":{"aliases":{}},".monitoring-kibana-6-2019.03.01":{"aliases":{}},"kibana_sample_data_logs":{"aliases":{}},"winsrv-2019.02.28":{"aliases":{}},"winsrv-2019.03.01":{"aliases":{}}}

root@elk:~# curl 'localhost:9200/_cat/indices?v'
health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .monitoring-kibana-6-2019.03.01 C1Um4omRSKmI15KjHKSbUg   1   0       9537            0      5.1mb          5.1mb
green  open   .monitoring-es-6-2019.02.28     g9347X4KRGa0g87PlwghRQ   1   0    3531448        27192      1.5gb          1.5gb
green  open   .monitoring-kibana-6-2019.02.28 -q4eivhsQDOxdeOv4nT93A   1   0      10388            0        3mb            3mb
green  open   .monitoring-kibana-6-2019.02.26 7fqpLp5wRfqvVHPDat-utA   1   0      11689            0      2.6mb          2.6mb
green  open   .monitoring-kibana-6-2019.02.27 6yRTvXYESkS53FQ3_5srrg   1   0      14961            0      3.4mb          3.4mb
green  open   .monitoring-es-6-2019.02.26     8-wy5QDXR1iXW-jHnH2e8w   1   0     416946        13902    203.3mb        203.3mb
green  open   .kibana_1                       2M9hIrowRU-Usn9l9y1Z_w   1   0         17            2     90.4kb         90.4kb
green  open   .monitoring-es-6-2019.03.01     HMvq-7pdQ06VrEXxye-f5A   1   0      59649          100     42.8mb         42.8mb
green  open   winsrv-2019.03.01               NE5l5-KKSpqfOnXtrjK97Q   5   0        121            0        1mb            1mb
green  open   winsrv-2019.02.28               hbCgWABGRHymlGE3WaBtJA   5   0     104396            0    127.5mb        127.5mb
green  open   .monitoring-es-6-2019.02.27     l4ueGl2HS3y2CEP7awCnfw   1   0    1151331        34196    503.2mb        503.2mb
green  open   kibana_sample_data_logs         EBRbGki-TC-Z1LLc-CwtOQ   1   0      14005            0     11.4mb         11.4mb

When yesterday i deleted indices they stayed in console for some time, but they was greyed out.

DavidTurner · March 1, 2019, 10:55am

By "developer console" I meant the thing on the left in Kibana labelled Dev Tools, but yes you can do this with curl too.

index                           pri rep docs.count docs.deleted store.size pri.store.size
.monitoring-kibana-6-2019.03.01   1   0       9537            0      5.1mb          5.1mb
.monitoring-es-6-2019.02.28       1   0    3531448        27192      1.5gb          1.5gb
.monitoring-kibana-6-2019.02.28   1   0      10388            0        3mb            3mb
.monitoring-kibana-6-2019.02.26   1   0      11689            0      2.6mb          2.6mb
.monitoring-kibana-6-2019.02.27   1   0      14961            0      3.4mb          3.4mb
.monitoring-es-6-2019.02.26       1   0     416946        13902    203.3mb        203.3mb
.kibana_1                         1   0         17            2     90.4kb         90.4kb
.monitoring-es-6-2019.03.01       1   0      59649          100     42.8mb         42.8mb
winsrv-2019.03.01                 5   0        121            0        1mb            1mb
winsrv-2019.02.28                 5   0     104396            0    127.5mb        127.5mb
.monitoring-es-6-2019.02.27       1   0    1151331        34196    503.2mb        503.2mb
kibana_sample_data_logs           1   0      14005            0     11.4mb         11.4mb

Ok, that adds up to ~5M documents and ~2.4GB mostly in .monitoring-es-6-2019.02.28 and .monitoring-es-6-2019.02.27 so the numbers in the UI look right to me. These count as "system indices" so I think if you switch that on in Kibana you will see details for these there too.

I note that you are using daily indices for your winsrv data and have left it with the default number_of_shards (i.e. 5, the pri column). These indices look quite small (so far) so you might prefer to use a longer time range and fewer shards to get each shard into the 20-40GB range and avoid the pitfalls of having too many shards:

KOTOXJle6 · March 1, 2019, 11:03am

Wow! That was pretty close and easy to find. But 5 millions of ES logs in one day is normal?

I have already changed shards value to 1

My Output:

output {
     if "winsrv" in [tags] {
        elasticsearch {
            hosts     => "localhost:9200"
            index    => "winsrv-%{+YYYY.MM.dd}"
             template => "/etc/logstash/conf.d/index_templates/winlogbeat.template.json"
             template_name => "winlogbeat"
             template_overwrite => true
        }
    }
    else {
        elasticsearch {
            hosts     => "localhost:9200"
            index    => "unknown_messages"
        }
    }

}

And /etc/logstash/conf.d/index_templates/winlogbeat.template.json ends with:

  "order": 1,
  "settings": {
    "index": {
      "mapping": {
        "total_fields": {
          "limit": 10000
        }
      },
      "number_of_shards": 1,
      "refresh_interval": "5s"
    }
  }
}

If i got it right, every new winsrv index will have 1 shard.

DavidTurner · March 1, 2019, 1:38pm

It's a little surprising to me too.

Looks like it; we will find out tomorrow

Maybe also consider monthly indices: winsrv-%{+YYYY.MM}. Depends on how much traffic you're anticipating, but yesterday's index was 127MB so if it carries on like that a whole month will only be ~4GB which is still pretty small.

system · March 29, 2019, 1:38pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.