Cluster management: 2000+ open active shards

Elastic Stack Elasticsearch
1

Hello,

I hope this messages finds you and your loved ones safe and healthy.

I have a 3 node Elastic cluster with 2 data nodes and 1 voting node. Nodes are running Ubuntu 20.04.2 LTS with the stack at the latest current release - 7.12.1. Primary purpose of the stack is to collect data from 30+ honeypots & I will be creating virtual networks and labs to emulate attacker behavior and reate detection rules under Elastic SIEM. I plan to use ML analysis to create anomaly detection of tunneling data for exfiltration.

I am fortunate enough to be provided with a license by Elastic to continue my research (thank you very much team Elastic). However, when I try to create ML jobs I get an error that 2000/2000 shards are active.

I have also lost the ability to monitor the cluster and get the following error:

[index_closed_exception] closed, with { index_uuid="OeFKiokSRviOkkjE9i12JA" & index="metricbeat-7.9.3-2021.02.21-000005" }: Check the Elasticsearch Monitoring cluster network connection or the load level of the nodes.

HTTP 400

**Edit -1: **
The cluster just failed with the following error:

index [.async-search] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];

Edit - 2:
I am unable to login and I am getting following error:

{"statusCode":429,"error":"Too Many Requests","message":"cluster_block_exception"}

Edit - 3:
I am getting following error while I try to start Kibana:

{"type":"log","@timestamp":"2021-05-01T14:12:04+00:00","tags":["fatal","root"],"pid":3188,"message":"Error: Unable to complete saved object migrations for the [.kibana_task_manager] index. Please check the health of your Elasticsearch cluster and try again. Error: [cluster_block_exception]: index [.kibana_task_manager_7.12.1_001] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];\n    at migrationStateActionMachine (/usr/share/kibana/src/core/server/saved_objects/migrationsv2/migrations_state_action_machine.js:138:13)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)\n    at async Promise.all (index 1)\n    at SavedObjectsService.start (/usr/share/kibana/src/core/server/saved_objects/saved_objects_service.js:163:7)\n    at Server.start (/usr/share/kibana/src/core/server/server.js:283:31)\n    at Root.start (/usr/share/kibana/src/core/server/root/index.js:58:14)\n    at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:100:5)\n    at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:169:5)"}

Edit - 4:

The cluster is yellow even though Elasticsearch is running on both the data and single voting node:

{
    "cluster_name": "data-analytics-1",
    "status": "yellow",
    "timed_out": false,
    "number_of_nodes": 3,
    "number_of_data_nodes": 2,
    "active_primary_shards": 1150,
    "active_shards": 1150,
    "relocating_shards": 0,
    "initializing_shards": 0,
    "unassigned_shards": 1150,
    "delayed_unassigned_shards": 0,
    "number_of_pending_tasks": 0,
    "number_of_in_flight_fetch": 0,
    "task_max_waiting_in_queue_millis": 0,
    "active_shards_percent_as_number": 50.0
}

Edit - 5:

I just saw that one of the node's HDD has gone to 95% usage. This is the primary node in the cluster. However, there has been no influx of logs to justify this. It was 89% at the start of the day.

I want to do housekeeping on the cluster since I will be using it for the next six months to finish my research at college. I expect around 2 TB of more data will be written to the cluster. I was hoping to get assistance in making the cluster usable beyond just the ingestion of data. Historial data is very important for my analysis.

Could I kindly request assistance from the community for cleaning up things on the cluster.

Thank you very much.

2

Can you run and post the following.

GET /_cat/nodes/?v&h=name,du,dt,dup,hp,hc,r

The error indicates that you have enter

disk usage exceeded flood-stage watermark

Meaning you have completely run out of disk space, and elasticsearch has gone into read-only mode to protect itself. You will need to choose what indices you want to clean up / delete or add more disk space or nodes.

After you clean up you will also need to reset your indices to read-write we can show you how but provide this and perhaps we can take a look.

you can also see all your indices with sorted by size with.

GET /_cat/indices/*?v&s=pri.store.size:desc

You also have a problem with unasigned shards, we will try to figure out that later.

1 Like
3

Thank you very much for helping me. :smiley: :smiley:

Hello, please find the diagnostic output:

name                du      dt   dup hp      hc r
primarynode      792gb 835.6gb 94.78 52   4.1gb cdfhilmrstw
secondarynode  739.5gb 835.6gb 88.50 68   5.4gb cdfhilmrstw
votingonlynode  15.6gb  96.4gb 16.19 33 169.5mb mv

The second cluster node represents the correct size as I checked the cluster in the morning and it was at ~89%.

Output of this exceeds the character limitation on the post. Hence please find it as a link:
users.ox.ac.uk/~kell4724/cluster_index.xlsx

Also there is one indice of 100 GB which isn't listed, I am not sure why :expressionless:

4

Hello, I see that logs have blown up to almost 100+ GB. Can I empty them?

image

5

  1. Unless you can add more storage you are going to need to clean up indices.
    There is no way you are going to stuff 2 more TB of data onto this cluster.
    So you need to decide what you are going Clean up or Add more storage.

  2. For some reason you replica shards are not getting assigned I am unclear why so at this point none of this data is replicated, meaning if you lose 1 node your data will be lost. (We can come back to that at some point.) This also means if you want a replica (HA / safety) it is going to require Twice the storage you have now. Again no way you are going to stuff more data onto this cluster.

  3. For the size of these nodes you have way too many shards, we can talk about fixing that at some point.

For the big index Can you just run this on that index I would like to see that too.

GET /_cat/indices/<big-indices-name>?v&s=pri.store.size:desc

Summary:
So you need to decide what you are going to clean up or add storage to the nodes.... Let us know what you decide....

6

You can empty logs if want (that is up to you and your admin) I would suspect that the logs are not on the same volume as the data but I supposed it could be. (they should not be)

7

So I am a student and this is my setup at homelab as part of my dissertation. I can copy /dev/null to the logs and empty them to get the shards to be at least assigned.

I will get back once I do that. BRB.

8

Quick question, do these logs have any relevance other then audit and diagnostics?

Can i just empty the folder?

I see it is 143 GB on the primary node:

image '

It is 86 GB on the secondary node:

image

9

They are logs... some people need them... some people don't it is your cluster if you don't need them clean them.

10

After you clean up space you can run this command and it will fix the read only indices, this is a bit of a hammer / global but should be fine for yours.

Then start cleaning up.

PUT /*/_settings
{
  "index": {
    "blocks.read_only_allow_delete": false,
    "blocks.read_only": false
  }
}

You still have a number of issues... all those tiny indices are just taking space and eating up JVM Heap. I will be AFK for a while check back later.

NOTE : Fixed

11

Thank you very much for the continuous help. Thank you.

I've deleted the logs and restarted the service. Waiting for it to settle as I see the cluster is in RED and it is initializing shards:

{
    "cluster_name": "data-analytics-1",
    "status": "red",
    "timed_out": false,
    "number_of_nodes": 3,
    "number_of_data_nodes": 2,
    "active_primary_shards": 1080,
    "active_shards": 1080,
    "relocating_shards": 0,
    "initializing_shards": 4,
    "unassigned_shards": 1216,
    "delayed_unassigned_shards": 0,
    "number_of_pending_tasks": 8,
    "number_of_in_flight_fetch": 0,
    "task_max_waiting_in_queue_millis": 145,
    "active_shards_percent_as_number": 46.95652173913044
}
12

I just notices that you have hundreds of closed indices as well... wow... you have a lot stuffed on this little cluster...

If you have more RAM on the Box you could increase the heap but not more than 28GB or 50% of the Host RAM. (note closed indices do not use heap)

I'll check back later or tomorrow.

1 Like
13

The status is now green:

{
    "cluster_name": "data-analytics-1",
    "status": "green",
    "timed_out": false,
    "number_of_nodes": 3,
    "number_of_data_nodes": 2,
    "active_primary_shards": 1150,
    "active_shards": 2300,
    "relocating_shards": 0,
    "initializing_shards": 0,
    "unassigned_shards": 0,
    "delayed_unassigned_shards": 0,
    "number_of_pending_tasks": 0,
    "number_of_in_flight_fetch": 0,
    "task_max_waiting_in_queue_millis": 0,
    "active_shards_percent_as_number": 100.0
}

Please let me know the tasks to carry out now.

14

Wow look at you :slight_smile:

Run those commands above to remove the read only state

Then run the cat nodes lets see where you are at with disk usage.

Then you need to clean up or decide what you want to do .. .clean up or add disk.

15

I have a workstation with 128 GB RAM. I've given 16 GB to each of the data nodes with 1 GB to voting node. I've allocated 3 vCPU per data node and 1 vCPU to the voting node. All of this is running on an ESXi cluster with Xeon W-1290 CPU having 10 physical and 20 virtual cores.

Yes the cluster has lot of telemetry data as part of my college research. This is my final year. This is when all of it comes together. :slight_smile:

16

So I would give 48GB to each Node and set the JVM to 24GB to each data node.

What does the disk space look like now .... can you get more Storage?

You still have ALOT of shards... you should combine some of those tiny indices into bigger indices that will make the whole cluster better...

1 Like
17

For the first one:

#! this request accesses system indices: [.apm-agent-configuration, .apm-custom-link, .async-search, .kibana_1, .kibana_2, .kibana_3, .kibana_4, .kibana_5, .kibana_6, .kibana_7, .kibana_7.12.0_001, .kibana_7.12.1_001, .kibana_task_manager_1, .kibana_task_manager_2, .kibana_task_manager_7.12.0_001, .kibana_task_manager_7.12.1_001, .reporting-2020-11-22, .reporting-2021-01-31, .reporting-2021-02-07, .reporting-2021-02-28, .reporting-2021-03-14, .reporting-2021-04-04, .reporting-2021-04-11, .reporting-2021-04-18, .reporting-2021-04-25, .security-7, .tasks, .transform-internal-005, .transform-internal-006], but in a future major version, direct access to system indices will be prevented by default
#! Overriding settings on system indices: [.transform-internal-*] -> [index.blocks.read_only, index.blocks.read_only_allow_delete], [.tasks*] -> [index.blocks.read_only, index.blocks.read_only_allow_delete], [.security-[0-9]+] -> [index.blocks.read_only, index.blocks.read_only_allow_delete]. This will not work in the next major version
{
  "acknowledged" : true
}

For the second one:

#! this request accesses system indices: [.apm-agent-configuration, .apm-custom-link, .async-search, .kibana_1, .kibana_2, .kibana_3, .kibana_4, .kibana_5, .kibana_6, .kibana_7, .kibana_7.12.0_001, .kibana_7.12.1_001, .kibana_security_session_1, .kibana_task_manager_1, .kibana_task_manager_2, .kibana_task_manager_7.12.0_001, .kibana_task_manager_7.12.1_001, .reporting-2020-11-22, .reporting-2021-01-31, .reporting-2021-02-07, .reporting-2021-02-28, .reporting-2021-03-14, .reporting-2021-04-04, .reporting-2021-04-11, .reporting-2021-04-18, .reporting-2021-04-25, .security-7, .tasks, .transform-internal-005, .transform-internal-006], but in a future major version, direct access to system indices will be prevented by default
#! Overriding settings on system indices: [.transform-internal-*] -> [index.blocks.read_only, index.blocks.read_only_allow_delete], [.tasks*] -> [index.blocks.read_only, index.blocks.read_only_allow_delete], [.security-[0-9]+] -> [index.blocks.read_only, index.blocks.read_only_allow_delete]. This will not work in the next major version
{
  "acknowledged" : true
}
18

Good I realize that the first one is only needed but good.

1 Like
19

  1. I will allocate the amount of RAM you've stated.

  2. I am from India and things are in lockdown but I will try and get a new SSD. It will be lot of work since I can't "add" addtional disk and I will have to clone and upgrade one of the 1TB disks to a 2TB disk. I can also move one of nodes to a NAS I have the IOPS will be a chellange. I will see what best can be done but getting the storage is a priority as I can't let go of 3 years of research in the final year :slight_smile: This is what all of the research was for :smiley:

  3. Last one I need help with. Could we connect for it in detail at per your convenience and availability?

20

Yes lockdown we are all worried about India, please be safe and take care!

No I am sorry I can not meet with you. I am a volunteer and it is community kinda against policy.

You are making great progress get to the next steps and and report back.

With respect to storage be carefull do you have this backed up anywhere?

Are you aware of Snapshot and Restore you could perhaps do that the NAS.

If you put 1 node on NAS and the other on SSD the whole cluster will run at NAS speed most likely but that could perhaps work too.

You can combined may of those smaller indices into larger with reindex API

What level license do you have?

Take Care / Good Luck.. I am heading AFK.

1 Like