# Failed to process cluster event (put-lifecycle-fb\_test) within 30s **URL:** \ **Category:** Elasticsearch\ **Created:** [March 26, 2024, 1:12pm UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-lifecycle-fb-test-within-30s/356180 "2024-03-26T13:12:48Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![tegerei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tegerei/32/132992_2.png) [@tegerei](https://discuss.elastic.co/u/tegerei)\ **Post date:** [March 26, 2024, 1:12pm UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-lifecycle-fb-test-within-30s/356180/1 "2024-03-26T13:12:49Z")
I am running a 3 node cluster for Elasticsearch. I get the error below when I try to create an ILM policy ```auto org.elasticsearch.transport.RemoteTransportException: [es3][192.168.10.52:9300][cluster:admin/ilm/put] Caused by: org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (put-lifecycle-fb_test) within 30s at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$0(MasterService.java:158) ~[elasticsearch-7.17.9.jar:7.17.9] at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?] at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$1(MasterService.java:157) ~[elasticsearch-7.17.9.jar:7.17.9] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) ~[elasticsearch-7.17.9.jar:7.17.9] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.lang.Thread.run(Thread.java:1589) [?:?] ``` The following is my cluster details: ```auto { "cluster_name" : "ess", "status" : "red", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 3, "active_primary_shards" : 2448, "active_shards" : 2503, "relocating_shards" : 0, "initializing_shards" : 4, "unassigned_shards" : 2120, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 134, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 1143201, "active_shards_percent_as_number" : 54.09552625891506 } ``` I have tried to restart the cluster, but every time I try to perform a synced flush, I get error 502 after 30 seconds. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 26, 2024, 2:57pm UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-lifecycle-fb-test-within-30s/356180/2 "2024-03-26T14:57:50Z")
The cluster is `red` so you might need to wait for the missing primary shards to be recovered. ---
**Author:** ![tegerei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tegerei/32/132992_2.png) [@tegerei](https://discuss.elastic.co/u/tegerei)\ **Post date:** [March 27, 2024, 11:52am UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-lifecycle-fb-test-within-30s/356180/3 "2024-03-27T11:52:55Z")
Hello, Even when the cluster health is Yellow, I still get the same error. Recently I also get these errors on the cluster. ```auto [2024-03-27T12:49:42,962][WARN][o.e.g.PersistedClusterStateService] [es3] writing cluster state took [15006ms] which is above the warn threshold of [10s]; wrote global metadata [false] and metadata for [4] indices and skipped [1598] unchanged indices ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 27, 2024, 11:57am UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-lifecycle-fb-test-within-30s/356180/4 "2024-03-27T11:57:48Z")
What is the hardware specification of the cluster? What type of storage are you using? ---
**Author:** ![tegerei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tegerei/32/132992_2.png) [@tegerei](https://discuss.elastic.co/u/tegerei)\ **Post date:** [March 27, 2024, 12:06pm UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-lifecycle-fb-test-within-30s/356180/5 "2024-03-27T12:06:09Z")
I have 3 nodes as follows: node1: 64 GB RAM, 32G heap size, 15 TB nvme SSD, 8 cores node2: 128GB RAM, 64GB heap size, 15 TB nvme SSD,12 cores node3: 128GB RAM, 64GB heap size, 15 TB nvme SSD, 12 cores All the nodes are master eligible. ---
**Author:** ![tegerei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tegerei/32/132992_2.png) [@tegerei](https://discuss.elastic.co/u/tegerei)\ **Post date:** [March 27, 2024, 12:11pm UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-lifecycle-fb-test-within-30s/356180/6 "2024-03-27T12:11:14Z")
Additional info ```auto shards disk.indices disk.used disk.avail disk.total disk.percent host ip node 1390 3.4tb 5.5tb 8.8tb 14.4tb 38 192.168.10.52 192.168.10.52 es1 192 103.1gb 3.3tb 11tb 14.4tb 23 192.168.10.81 192.168.10.81 es2 1344 5tb 6.1tb 8.3tb 14.4tb 42 192.168.10.25 192.168.10.25 es3 1703 UNASSIGNED ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 24, 2024, 12:12pm UTC](https://discuss.elastic.co/t/failed-to-process-cluster-event-put-lifecycle-fb-test-within-30s/356180/7 "2024-04-24T12:12:01Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.