Cluster update failure message (Data size issue?)

Alexandros888 · April 24, 2020, 7:27am

Hello,

I have a 3 nodes cluster and yesterday was time for updating them.

During the end of the 1st node update procedure when my cluster was at the following state:

I received that message when i tried to start kibana in that node from kibana service inside the node:

Nevertheless the issue was fixed by its own later when all 3 nodes where available again as the mage show below:

Was that a data size issue? More specifically when i shut down the first node did all its stored data try to be handeld by the 2 other nodes but the size was so big that they couldnt?

I have 3 nodes with 500GB each , so when i shut down one of them the other 2 tried to take his data load (291.6 + 337.8 + 375.5 =1004.9>1000) and that led to failure?

Finnaly, should we always choose to start from the master node for the cluster update or it does not make any difference?

Thank you alot

warkolm · April 27, 2020, 12:11am

Please don't post pictures of text, they are difficult to read, impossible to search and some people may not be even able to see them

Alexandros888 · April 27, 2020, 6:29am

Hello,

Thank you for the hint :). I also attach now the relative logs that the image included:

(The last line of the logs below include the FATAL ERROR)

kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Thu 2020-04-23 15:26:22 CEST; 6min ago
  Process: 5695 ExecStart=/usr/share/kibana/bin/kibana -c /etc/kibana/kibana.yml (code=exited, status=1/FAILURE)
Main PID: 5695 (code=exited, status=1/FAILURE)

Apr 23 15:26:19 sag-prd-es-003.sag.services systemd[1]: Unit kibana.service entered failed state.
Apr 23 15:26:19 sag-prd-es-003.sag.services systemd[1]: kibana.service failed.
Apr 23 15:26:22 sag-prd-es-003.sag.services systemd[1]: kibana.service holdoff time over, scheduling restart.
Apr 23 15:26:22 sag-prd-es-003.sag.services systemd[1]: Stopped Kibana.
Apr 23 15:26:22 sag-prd-es-003.sag.services systemd[1]: start request repeated too quickly for kibana.service
Apr 23 15:26:22 sag-prd-es-003.sag.services systemd[1]: Failed to start Kibana.
Apr 23 15:26:22 sag-prd-es-003.sag.services systemd[1]: Unit kibana.service entered failed state.
Apr 23 15:26:22 sag-prd-es-003.sag.services systemd[1]: kibana.service failed.
[root@sag-prd-es-003 log]# journalctl -fu kibana.service
-- Logs begin at Thu 2020-04-23 14:46:02 CEST. --
Apr 23 15:26:18 sag-prd-es-003.sag.services kibana[5695]: FATAL  [illegal_argument_exception] Validation Failed: 1: this action would add [2] total shards, but this cluster curr               ently has [2252]/[2000] maximum shards open; :: {"path":"/.kibana_task_manager_2","query":{},"body":"{\"mappings\":{\"dynamic\":\"strict\",\"properties\":{\"kibana\":{\"properti               es\":{\"apiVersion\":{\"type\":\"integer\"},\"uuid\":{\"type\":\"keyword\"},\"version\":{\"type\":\"integer\"}}},\"task\":{\"properties\":{\"taskType\":{\"type\":\"keyword\"},\"               scheduledAt\":{\"type\":\"date\"},\"runAt\":{\"type\":\"date\"},\"startedAt\":{\"type\":\"date\"},\"retryAt\":{\"type\":\"date\"},\"schedule\":{\"properties\":{\"interval\":{\"t               ype\":\"keyword\"}}},\"attempts\":{\"type\":\"integer\"},\"status\":{\"type\":\"keyword\"},\"params\":{\"type\":\"text\"},\"state\":{\"type\":\"text\"},\"user\":{\"type\":\"keyw               ord\"},\"scope\":{\"type\":\"keyword\"},\"ownerId\":{\"type\":\"keyword\"}}},\"type\":{\"type\":\"keyword\"},\"config\":{\"dynamic\":\"true\",\"properties\":{\"buildNum\":{\"typ               e\":\"keyword\"}}},\"migrationVersion\":{\"dynamic\":\"true\",\"type\":\"object\"},\"namespace\":{\"type\":\"keyword\"},\"updated_at\":{\"type\":\"date\"},\"references\":{\"type               \":\"nested\",\"properties\":{\"name\":{\"type\":\"keyword\"},\"type\":{\"type\":\"keyword\"},\"id\":{\"type\":\"keyword\"}}}},\"_meta\":{\"migrationMappingPropertyHashes\":{\"c               onfig\":\"87aca8fdb053154f11383fce3dbf3edf\",\"migrationVersion\":\"4a1746014a75ade3a714e1db5763276f\",\"type\":\"2f4316de49999235636386fe51dc06c1\",\"namespace\":\"2f4316de4999               9235636386fe51dc06c1\",\"updated_at\":\"00da57df13e94e9d98437d13ace4bfe0\",\"references\":\"7997cf5a56cc02bdc9c93361bde732b0\",\"task\":\"235412e52d09e7165fac8a67a43ad6b4\"}}},\               "settings\":{\"number_of_shards\":1,\"auto_expand_replicas\":\"0-1\"}}","statusCode":400,"response":"{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reaso               n\":\"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [2252]/[2000] maximum shards open;\"}],\"type\":\"illegal_argument_exception\"               ,\"reason\":\"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [2252]/[2000] maximum shards open;\"},\"status\":400}"}

So as it is noticed from the logs above it says: FATAL [illegal_argument_exception] Validation Failed: 1: this action would add [2] total shards, but this cluster curr ently has [2252]/[2000] maximum shards open; ::

Was that a data size issue? More specifically when i shut down the first node did all its stored data try to be handeld by the 2 other nodes but the size was so big that they couldnt??

I have 3 nodes with 500GB each , so when i shut down one of them the other 2 tried to take his data load (291.6 + 337.8 + 375.5 =1004.9>1000) and that led to failure?

warkolm · April 27, 2020, 7:05am

What does the output from _cat/nodes look like?

Alexandros888 · April 27, 2020, 7:13am

It is exactly like that:

> 10.1.163.50 63 99  8 0.83 1.05 1.28 dilm - sag-prd-es-001.sag.services
> 10.1.163.52 37 98 14 1.35 1.70 1.82 dilm * sag-prd-es-003.sag.services
> 10.1.163.51 19 98 10 1.20 1.04 1.24 dilm - sag-prd-es-002.sag.services

warkolm · April 27, 2020, 7:15am

Thanks, what about _cat/allocation?v.

Alexandros888 · April 27, 2020, 7:18am

Hello,

It is like that:

shards disk.indices disk.used disk.avail disk.total disk.percent host        ip          node
   762      361.5gb   364.4gb    135.2gb    499.7gb           72 10.1.163.52 10.1.163.52 sag-prd-es-003.sag.services
   762        321gb     324gb    175.7gb    499.7gb           64 10.1.163.51 10.1.163.51 sag-prd-es-002.sag.services
   762      328.3gb   330.1gb    169.6gb    499.7gb           66 10.1.163.50 10.1.163.50 sag-prd-es-001.sag.services

Thank you

warkolm · April 27, 2020, 7:23am

Cool, thanks. Don't forget it's heaps easier to read that sort of output if you wrap it with the code tags

High level, you have too many shards for that size data. You could look at using the _shrink endpoint to reduce that, which will help this issue, but also more generally make better use of your cluster's resources. Then look at ILM to make that pretty much a non-existent issue moving forward!

Short term you can use the _close endpoint to get around the 2000 limit the logs mention. But it's really a short term work around, not a fix by any means.

Alexandros888 · April 27, 2020, 8:08am

Hello,

Thank you very much for the valuable info shared. The recommendations are very interesting indeed and will be followed.

So in order to see if i understood well my issue i had 2 problems:

1st problem: I exceeded the shard limit size that is probably 1000 per node. So when i shut down 1 node the other 2 nodes that have together shard limit 1000+1000=2000 could not handle 762+762+762=2286 shards because 2286>2000 and so a FATAL error occurred.

2nd problem: I have 3 nodes with 500GB each , so when i shut down one of them the other 2 tried to take his data load (291.6 + 337.8 + 375.5 =1004.9>1000) and that led to data size failure as well.

Correct?
Thank you

warkolm · April 27, 2020, 8:14am

Elasticsearch has a soft limit, that you can increase. But Kibana is a little more hard on taking that limit as it is.

It'll try to balance the shards across both nodes, yes. That will likely also have attributed somewhat to the issue.

Alexandros888 · April 27, 2020, 8:26am

Thank you very much. !!!!!!

warkolm · April 27, 2020, 8:37am

No worries. Good luck with it and thanks for listening to the code formatting request, it really does make it heaps easier for us to help

Alexandros888 · April 27, 2020, 9:11am

Welcome :).

Also i question that was in my first comment but i forgot to mention it again:

When we do a cluster upgrade should we always choose to start our upgrade from the master node and repeat the procedure for the remaining nodes depending on who is the new master node or it does not make any difference?

Note: all my 3 nodes are master eligible nodes in my cluster.

warkolm · April 27, 2020, 9:16am

I would do your master last, so that you should only have to have one re-election process.

Alexandros888 · April 27, 2020, 9:32am

Makes total sense thank you again

system · May 25, 2020, 9:32am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.