# Cluster update failure message (Data size issue?) **URL:** \ **Category:** Elasticsearch\ **Created:** [April 24, 2020, 7:27am UTC](https://discuss.elastic.co/t/cluster-update-failure-message-data-size-issue/229588 "2020-04-24T07:27:53Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![Alexandros888](https://avatars.discourse-cdn.com/v4/letter/a/3ec8ea/32.png) [@Alexandros888](https://discuss.elastic.co/u/Alexandros888)\ **Post date:** [April 24, 2020, 7:27am UTC](https://discuss.elastic.co/t/cluster-update-failure-message-data-size-issue/229588/1 "2020-04-24T07:27:53Z")
Hello, I have a 3 nodes cluster and yesterday was time for updating them. During the end of the 1st node update procedure when my cluster was at the following state: ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/0/00a30c65ba4019759d19acef64819f77e7b3524c.png) I received that message when i tried to start kibana in that node from kibana service inside the node: ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/e/7ed98aaadecc61d7965727faae24cd3ed5ea5729.png) Nevertheless the issue was fixed by its own later when all 3 nodes where available again as the mage show below: ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/0/d0db49f913e48d21e0977bd609e5d8ae176973f3.png) Was that a data size issue? More specifically when i shut down the first node did all its stored data try to be handeld by the 2 other nodes but the size was so big that they couldnt? I have 3 nodes with 500GB each , so when i shut down one of them the other 2 tried to take his data load (291.6 + 337.8 + 375.5 =1004.9\>1000) and that led to failure? Finnaly, should we always choose to start from the master node for the cluster update or it does not make any difference? Thank you alot ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 27, 2020, 12:11am UTC](https://discuss.elastic.co/t/cluster-update-failure-message-data-size-issue/229588/2 "2020-04-27T00:11:45Z")
Please don't post pictures of text, they are difficult to read, impossible to search and some people may not be even able to see them 🙂 ---
**Author:** ![Alexandros888](https://avatars.discourse-cdn.com/v4/letter/a/3ec8ea/32.png) [@Alexandros888](https://discuss.elastic.co/u/Alexandros888)\ **Post date:** [April 27, 2020, 6:29am UTC](https://discuss.elastic.co/t/cluster-update-failure-message-data-size-issue/229588/3 "2020-04-27T06:29:55Z")
Hello, Thank you for the hint :). I also attach now the relative logs that the image included: (The last line of the logs below include the FATAL ERROR) ``` kibana.service - Kibana Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled) Active: failed (Result: start-limit) since Thu 2020-04-23 15:26:22 CEST; 6min ago Process: 5695 ExecStart=/usr/share/kibana/bin/kibana -c /etc/kibana/kibana.yml (code=exited, status=1/FAILURE) Main PID: 5695 (code=exited, status=1/FAILURE) Apr 23 15:26:19 sag-prd-es-003.sag.services systemd[1]: Unit kibana.service entered failed state. Apr 23 15:26:19 sag-prd-es-003.sag.services systemd[1]: kibana.service failed. Apr 23 15:26:22 sag-prd-es-003.sag.services systemd[1]: kibana.service holdoff time over, scheduling restart. Apr 23 15:26:22 sag-prd-es-003.sag.services systemd[1]: Stopped Kibana. Apr 23 15:26:22 sag-prd-es-003.sag.services systemd[1]: start request repeated too quickly for kibana.service Apr 23 15:26:22 sag-prd-es-003.sag.services systemd[1]: Failed to start Kibana. Apr 23 15:26:22 sag-prd-es-003.sag.services systemd[1]: Unit kibana.service entered failed state. Apr 23 15:26:22 sag-prd-es-003.sag.services systemd[1]: kibana.service failed. [root@sag-prd-es-003 log]# journalctl -fu kibana.service -- Logs begin at Thu 2020-04-23 14:46:02 CEST. -- Apr 23 15:26:18 sag-prd-es-003.sag.services kibana[5695]: FATAL [illegal_argument_exception] Validation Failed: 1: this action would add [2] total shards, but this cluster curr ently has [2252]/[2000] maximum shards open; :: {"path":"/.kibana_task_manager_2","query":{},"body":"{\"mappings\":{\"dynamic\":\"strict\",\"properties\":{\"kibana\":{\"properti es\":{\"apiVersion\":{\"type\":\"integer\"},\"uuid\":{\"type\":\"keyword\"},\"version\":{\"type\":\"integer\"}}},\"task\":{\"properties\":{\"taskType\":{\"type\":\"keyword\"},\" scheduledAt\":{\"type\":\"date\"},\"runAt\":{\"type\":\"date\"},\"startedAt\":{\"type\":\"date\"},\"retryAt\":{\"type\":\"date\"},\"schedule\":{\"properties\":{\"interval\":{\"t ype\":\"keyword\"}}},\"attempts\":{\"type\":\"integer\"},\"status\":{\"type\":\"keyword\"},\"params\":{\"type\":\"text\"},\"state\":{\"type\":\"text\"},\"user\":{\"type\":\"keyw ord\"},\"scope\":{\"type\":\"keyword\"},\"ownerId\":{\"type\":\"keyword\"}}},\"type\":{\"type\":\"keyword\"},\"config\":{\"dynamic\":\"true\",\"properties\":{\"buildNum\":{\"typ e\":\"keyword\"}}},\"migrationVersion\":{\"dynamic\":\"true\",\"type\":\"object\"},\"namespace\":{\"type\":\"keyword\"},\"updated_at\":{\"type\":\"date\"},\"references\":{\"type \":\"nested\",\"properties\":{\"name\":{\"type\":\"keyword\"},\"type\":{\"type\":\"keyword\"},\"id\":{\"type\":\"keyword\"}}}},\"_meta\":{\"migrationMappingPropertyHashes\":{\"c onfig\":\"87aca8fdb053154f11383fce3dbf3edf\",\"migrationVersion\":\"4a1746014a75ade3a714e1db5763276f\",\"type\":\"2f4316de49999235636386fe51dc06c1\",\"namespace\":\"2f4316de4999 9235636386fe51dc06c1\",\"updated_at\":\"00da57df13e94e9d98437d13ace4bfe0\",\"references\":\"7997cf5a56cc02bdc9c93361bde732b0\",\"task\":\"235412e52d09e7165fac8a67a43ad6b4\"}}},\ "settings\":{\"number_of_shards\":1,\"auto_expand_replicas\":\"0-1\"}}","statusCode":400,"response":"{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reaso n\":\"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [2252]/[2000] maximum shards open;\"}],\"type\":\"illegal_argument_exception\" ,\"reason\":\"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [2252]/[2000] maximum shards open;\"},\"status\":400}"} ``` So as it is noticed from the logs above it says: **FATAL [illegal\_argument\_exception] Validation Failed: 1: this action would add [2] total shards, but this cluster curr ently has [2252]/[2000] maximum shards open; ::** Was that a data size issue? More specifically when i shut down the first node did all its stored data try to be handeld by the 2 other nodes but the size was so big that they couldnt?? I have 3 nodes with 500GB each , so when i shut down one of them the other 2 tried to take his data load (291.6 + 337.8 + 375.5 =1004.9\>1000) and that led to failure? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 27, 2020, 7:05am UTC](https://discuss.elastic.co/t/cluster-update-failure-message-data-size-issue/229588/4 "2020-04-27T07:05:07Z")
What does the output from `_cat/nodes` look like? ---
**Author:** ![Alexandros888](https://avatars.discourse-cdn.com/v4/letter/a/3ec8ea/32.png) [@Alexandros888](https://discuss.elastic.co/u/Alexandros888)\ **Post date:** [April 27, 2020, 7:13am UTC](https://discuss.elastic.co/t/cluster-update-failure-message-data-size-issue/229588/5 "2020-04-27T07:13:08Z")
> [@warkolm](#): > > \_cat/nodes It is exactly like that: ``` > 10.1.163.50 63 99 8 0.83 1.05 1.28 dilm - sag-prd-es-001.sag.services > 10.1.163.52 37 98 14 1.35 1.70 1.82 dilm * sag-prd-es-003.sag.services > 10.1.163.51 19 98 10 1.20 1.04 1.24 dilm - sag-prd-es-002.sag.services ``` ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 27, 2020, 7:15am UTC](https://discuss.elastic.co/t/cluster-update-failure-message-data-size-issue/229588/6 "2020-04-27T07:15:41Z")
Thanks, what about `_cat/allocation?v`. ---
**Author:** ![Alexandros888](https://avatars.discourse-cdn.com/v4/letter/a/3ec8ea/32.png) [@Alexandros888](https://discuss.elastic.co/u/Alexandros888)\ **Post date:** [April 27, 2020, 7:18am UTC](https://discuss.elastic.co/t/cluster-update-failure-message-data-size-issue/229588/7 "2020-04-27T07:18:22Z")
> [@warkolm](#): > > \_cat/allocation?v Hello, It is like that: ``` shards disk.indices disk.used disk.avail disk.total disk.percent host ip node 762 361.5gb 364.4gb 135.2gb 499.7gb 72 10.1.163.52 10.1.163.52 sag-prd-es-003.sag.services 762 321gb 324gb 175.7gb 499.7gb 64 10.1.163.51 10.1.163.51 sag-prd-es-002.sag.services 762 328.3gb 330.1gb 169.6gb 499.7gb 66 10.1.163.50 10.1.163.50 sag-prd-es-001.sag.services ``` Thank you ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 27, 2020, 7:23am UTC](https://discuss.elastic.co/t/cluster-update-failure-message-data-size-issue/229588/8 "2020-04-27T07:23:08Z")
Cool, thanks. Don't forget it's heaps easier to read that sort of output if you wrap it with the code tags 😃 High level, you have too many shards for that size data. You could look at using the `_shrink` endpoint to reduce that, which will help this issue, but also more generally make better use of your cluster's resources. Then look at [ILM](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/getting-started-index-lifecycle-management.html) to make that pretty much a non-existent issue moving forward! 🙂 Short term you can use the [`_close`](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/indices-close.html) endpoint to get around the 2000 limit the logs mention. But it's really a short term work around, not a fix by any means. ---
**Author:** ![Alexandros888](https://avatars.discourse-cdn.com/v4/letter/a/3ec8ea/32.png) [@Alexandros888](https://discuss.elastic.co/u/Alexandros888)\ **Post date:** [April 27, 2020, 8:08am UTC](https://discuss.elastic.co/t/cluster-update-failure-message-data-size-issue/229588/9 "2020-04-27T08:08:07Z")
Hello, Thank you very much for the valuable info shared. The recommendations are very interesting indeed and will be followed. So in order to see if i understood well my issue i had 2 problems: **1st problem:** I exceeded the shard limit size that is probably 1000 per node. So when i shut down 1 node the other 2 nodes that have together shard limit 1000+1000=2000 could not handle 762+762+762=2286 shards because 2286\>2000 and so a FATAL error occurred. **2nd problem:** I have 3 nodes with 500GB each , so when i shut down one of them the other 2 tried to take his data load (291.6 + 337.8 + 375.5 =1004.9\>1000) and that led to data size failure as well. Correct? Thank you ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 27, 2020, 8:14am UTC](https://discuss.elastic.co/t/cluster-update-failure-message-data-size-issue/229588/10 "2020-04-27T08:14:28Z")
> [@Alexandros888](#): > > 1st problem Elasticsearch has a soft limit, that you can increase. But Kibana is a little more hard on taking that limit as it is. > [@Alexandros888](#): > > 2nd problem It'll try to balance the shards across both nodes, yes. That will likely also have attributed somewhat to the issue. ---
**Author:** ![Alexandros888](https://avatars.discourse-cdn.com/v4/letter/a/3ec8ea/32.png) [@Alexandros888](https://discuss.elastic.co/u/Alexandros888)\ **Post date:** [April 27, 2020, 8:26am UTC](https://discuss.elastic.co/t/cluster-update-failure-message-data-size-issue/229588/11 "2020-04-27T08:26:50Z")
Thank you very much. !!!!!! 🙂 ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 27, 2020, 8:37am UTC](https://discuss.elastic.co/t/cluster-update-failure-message-data-size-issue/229588/12 "2020-04-27T08:37:47Z")
No worries. Good luck with it and thanks for listening to the `code` formatting request, it really does make it heaps easier for us to help 🙂 ---
**Author:** ![Alexandros888](https://avatars.discourse-cdn.com/v4/letter/a/3ec8ea/32.png) [@Alexandros888](https://discuss.elastic.co/u/Alexandros888)\ **Post date:** [April 27, 2020, 9:11am UTC](https://discuss.elastic.co/t/cluster-update-failure-message-data-size-issue/229588/13 "2020-04-27T09:11:39Z")
Welcome :). Also i question that was in my first comment but i forgot to mention it again: When we do a cluster upgrade should we always choose to start our upgrade from the master node and repeat the procedure for the remaining nodes depending on who is the new master node or it does not make any difference? Note: all my 3 nodes are master eligible nodes in my cluster. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 27, 2020, 9:16am UTC](https://discuss.elastic.co/t/cluster-update-failure-message-data-size-issue/229588/14 "2020-04-27T09:16:55Z")
I would do your master last, so that you should only have to have one re-election process. ---
**Author:** ![Alexandros888](https://avatars.discourse-cdn.com/v4/letter/a/3ec8ea/32.png) [@Alexandros888](https://discuss.elastic.co/u/Alexandros888)\ **Post date:** [April 27, 2020, 9:32am UTC](https://discuss.elastic.co/t/cluster-update-failure-message-data-size-issue/229588/15 "2020-04-27T09:32:33Z")
Makes total sense thank you again 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 25, 2020, 9:32am UTC](https://discuss.elastic.co/t/cluster-update-failure-message-data-size-issue/229588/16 "2020-05-25T09:32:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.