# Logging-and-metrics stuck **URL:** \ **Category:** Elastic Cloud Enterprise (ECE)\ **Created:** [May 24, 2018, 4:07pm UTC](https://discuss.elastic.co/t/logging-and-metrics-stuck/133191 "2018-05-24T16:07:11Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![jmtrc](https://avatars.discourse-cdn.com/v4/letter/j/c37758/32.png) [@jmtrc](https://discuss.elastic.co/u/jmtrc)\ **Post date:** [May 24, 2018, 4:07pm UTC](https://discuss.elastic.co/t/logging-and-metrics-stuck/133191/1 "2018-05-24T16:07:12Z")
The cluster looks healthy: { "cluster\_name" : "05274b09effb4f9f9d237d0f92204ce9", "status" : "green", "timed\_out" : false, "number\_of\_nodes" : 7, "number\_of\_data\_nodes" : 7, "active\_primary\_shards" : 63, "active\_shards" : 70, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 0, "delayed\_unassigned\_shards" : 0, "number\_of\_pending\_tasks" : 0, "number\_of\_in\_flight\_fetch" : 0, "task\_max\_waiting\_in\_queue\_millis" : 0, "active\_shards\_percent\_as\_number" : 100.0 } But when I was trying to increase size from 1GB to 2GB, it seems stuck with messages like these: [2018-05-24 16:04:43,380][INFO][no.found.cluster.plan.elasticsearch.05274b09effb4f9f9d237d0f92204ce9] Timeout waiting for cluster states to update, waiting a little before rechecking... {"cluster\_id":"05274b09effb4f9f9d237d0f92204ce9"} [2018-05-24 16:04:53,497][INFO][no.found.cluster.plan.elasticsearch.05274b09effb4f9f9d237d0f92204ce9] Setting settings: [HttpRequest(PUT,/\_cluster/settings,List(),HttpEntity(application/json; charset=UTF-8,{"transient":{"cluster":{"routing":{"allocation":{"exclude":{"\_name":"instance-0000000001,1527177893494"},"awareness":{"attributes":""}}}}}}),HTTP/1.1)] {"cluster\_id":"05274b09effb4f9f9d237d0f92204ce9"} [2018-05-24 16:04:53,655][INFO][no.found.cluster.plan.elasticsearch.05274b09effb4f9f9d237d0f92204ce9] Found bad verifications: [List((ElasticsearchInstance(ElasticsearchCluster(05274b09effb4f9f9d237d0f92204ce9),instance-0000000000),Missing(ElasticsearchInstance(ElasticsearchCluster(05274b09effb4f9f9d237d0f92204ce9),instance-0000000000)),no.found.curator.pimps.FutureWatchedEvent@6df3d48e))] {"cluster\_id":"05274b09effb4f9f9d237d0f92204ce9"} [2018-05-24 16:04:53,655][INFO][no.found.cluster.plan.elasticsearch.05274b09effb4f9f9d237d0f92204ce9] Verified migration with result: [List((ElasticsearchInstance(ElasticsearchCluster(05274b09effb4f9f9d237d0f92204ce9),instance-0000000006),ClusterStateVerified(true,QEw\_XswtQxaT4y22ulhZrw,Set(STARTED)),no.found.curator.pimps.FutureWatchedEvent@1e62aa77), (ElasticsearchInstance(ElasticsearchCluster(05274b09effb4f9f9d237d0f92204ce9),instance-0000000007),ClusterStateVerified(true,QEw\_XswtQxaT4y22ulhZrw,Set(STARTED)),no.found.curator.pimps.FutureWatchedEvent@1b323d68), (ElasticsearchInstance(ElasticsearchCluster(05274b09effb4f9f9d237d0f92204ce9),instance-0000000004),ClusterStateVerified(true,QEw\_XswtQxaT4y22ulhZrw,Set(STARTED)),no.found.curator.pimps.FutureWatchedEvent@75722552), (ElasticsearchInstance(ElasticsearchCluster(05274b09effb4f9f9d237d0f92204ce9),instance-0000000005),ClusterStateVerified(true,QEw\_XswtQxaT4y22ulhZrw,Set(STARTED)),no.found.curator.pimps.FutureWatchedEvent@826be57), (ElasticsearchInstance(ElasticsearchCluster(05274b09effb4f9f9d237d0f92204ce9),instance-0000000002),ClusterStateVerified(true,QEw\_XswtQxaT4y22ulhZrw,Set(STARTED)),no.found.curator.pimps.FutureWatchedEvent@31c8c36), (ElasticsearchInstance(ElasticsearchCluster(05274b09effb4f9f9d237d0f92204ce9),instance-0000000003),ClusterStateVerified(true,QEw\_XswtQxaT4y22ulhZrw,Set(STARTED)),no.found.curator.pimps.FutureWatchedEvent@94f3840), (ElasticsearchInstance(ElasticsearchCluster(05274b09effb4f9f9d237d0f92204ce9),instance-0000000000),Missing(ElasticsearchInstance(ElasticsearchCluster(05274b09effb4f9f9d237d0f92204ce9),instance-0000000000)),no.found.curator.pimps.FutureWatchedEvent@6df3d48e), (ElasticsearchInstance(ElasticsearchCluster(05274b09effb4f9f9d237d0f92204ce9),instance-0000000001),ClusterStateVerified(true,QEw\_XswtQxaT4y22ulhZrw,Set(STARTED)),no.found.curator.pimps.FutureWatchedEvent@509a1b61))] {"cluster\_id":"05274b09effb4f9f9d237d0f92204ce9"} [2018-05-24 16:04:53,494][INFO][no.found.cluster.plan.elasticsearch.05274b09effb4f9f9d237d0f92204ce9] Need to set exclusions and awareness attributes. Current values are [Set(instance-0000000001, 1527177873195)] and [Set()] {"cluster\_id":"05274b09effb4f9f9d237d0f92204ce9"} I was watching /\_cat/shards for anything not STARTED but all shards were reported STARTED. Does this mean anything? Thanks. Jin. ---
**Author:** ![jmtrc](https://avatars.discourse-cdn.com/v4/letter/j/c37758/32.png) [@jmtrc](https://discuss.elastic.co/u/jmtrc)\ **Post date:** [May 24, 2018, 6:00pm UTC](https://discuss.elastic.co/t/logging-and-metrics-stuck/133191/2 "2018-05-24T18:00:42Z")
Also noticed that this is a cluster with 1 zone and 1 node. But there are 7 nodes reported. I tried to remove replicas by setting number\_of\_replicas to 0. But only one node is vacant with no data per ECE admin console. Other nodes all reported some data. Feeling like something messed up in previous attempts to increase the memory for this cluster. ---
**Author:** ![jmtrc](https://avatars.discourse-cdn.com/v4/letter/j/c37758/32.png) [@jmtrc](https://discuss.elastic.co/u/jmtrc)\ **Post date:** [May 25, 2018, 7:45pm UTC](https://discuss.elastic.co/t/logging-and-metrics-stuck/133191/3 "2018-05-25T19:45:27Z")
Under the same ECE adminstration, I noticed another more important 2x2 cluster reports 6 nodes plus one tiebreaker node. By looking at constructor logs, I see similar entries like: ``` [2018-05-25 12:47:03,587][INFO][no.found.cluster.plan.elasticsearch.40215dec0d49463cac59ab45379c8478] Need to set exclusions and awareness attributes. Current values are [Set(instance-0000000028, instance-0000000027, 1527252403592)] and [Set()] {"cluster_id":"40215dec0d49463cac59ab45379c8478"} [2018-05-25 12:47:07,799][INFO][no.found.cluster.plan.elasticsearch.40215dec0d49463cac59ab45379c8478] Found bad verifications: [List((ElasticsearchInstance(ElasticsearchCluster(40215dec0d49463cac59ab45379c8478),instance-0000000028),ClusterStateVerified(false,9-rCAmBSRoKuo1pAJ-bt2g,Set(STARTED)),no.found.curator.pimps.FutureWatchedEvent@1c5c72ae), (ElasticsearchInstance(ElasticsearchCluster(40215dec0d49463cac59ab45379c8478),instance-0000000039),ClusterStateVerified(false,9-rCAmBSRoKuo1pAJ-bt2g,Set(STARTED)),no.found.curator.pimps.FutureWatchedEvent@10860212), (ElasticsearchInstance(ElasticsearchCluster(40215dec0d49463cac59ab45379c8478),instance-0000000037),ClusterStateVerified(false,9-rCAmBSRoKuo1pAJ-bt2g,Set(STARTED)),no.found.curator.pimps.FutureWatchedEvent@5c4f6624), (ElasticsearchInstance(ElasticsearchCluster(40215dec0d49463cac59ab45379c8478),instance-0000000027),ClusterStateVerified(false,9-rCAmBSRoKuo1pAJ-bt2g,Set(STARTED)),no.found.curator.pimps.FutureWatchedEvent@14e0d97c), (ElasticsearchInstance(ElasticsearchCluster(40215dec0d49463cac59ab45379c8478),instance-0000000038),ClusterStateVerified(false,9-rCAmBSRoKuo1pAJ-bt2g,Set(STARTED)),no.found.curator.pimps.FutureWatchedEvent@6bd6004c), (ElasticsearchInstance(ElasticsearchCluster(40215dec0d49463cac59ab45379c8478),instance-0000000040),ClusterStateVerified(false,9-rCAmBSRoKuo1pAJ-bt2g,Set(STARTED)),no.found.curator.pimps.FutureWatchedEvent@4b6009b6), (ElasticsearchInstance(ElasticsearchCluster(40215dec0d49463cac59ab45379c8478),tiebreaker-0000000036),ClusterStateVerified(false,9-rCAmBSRoKuo1pAJ-bt2g,Set(STARTED)),no.found.curator.pimps.FutureWatchedEvent@6cb938b8))] {"cluster_id":"40215dec0d49463cac59ab45379c8478"} ``` This happened after an attempt to move nodes from one runner to another one using admin console. Any suggestions to sort this out? Thank you. Jin. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 8, 2018, 7:45pm UTC](https://discuss.elastic.co/t/logging-and-metrics-stuck/133191/4 "2018-06-08T19:45:43Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.