# AlreadyClosedException: engine is closed during cluster stats **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-monitoring\ **Created:** [January 7, 2019, 3:55pm UTC](https://discuss.elastic.co/t/alreadyclosedexception-engine-is-closed-during-cluster-stats/163238 "2019-01-07T15:55:18Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![katre](https://avatars.discourse-cdn.com/v4/letter/k/f6c823/32.png) [@katre](https://discuss.elastic.co/u/katre)\ **Post date:** [January 7, 2019, 3:55pm UTC](https://discuss.elastic.co/t/alreadyclosedexception-engine-is-closed-during-cluster-stats/163238/1 "2019-01-07T15:55:18Z")
We monitor elasticsearch by using -cluster/stats. However it seems to give false alerts or we have some underlying issue to be solved. Cluster contains of 3 nodes. We use curator to periodically delete old indexes and roll indexes. Elasticsearch seems to work fine and curator logs have no errors. On rare occasion during cluster stats there is an exception: [2018-12-15T15:30:03,719][INFO][o.e.c.r.a.AllocationService] [gdph-es-01] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[gh-partition-000592][1]] ...]). [2018-12-15T15:40:03,403][INFO][o.e.c.m.MetaDataDeleteIndexService] [gdph-es-01] [gh-partition-000538/mY4H42tjQsS72\_WEomcpzw] deleting index [2018-12-15T16:30:04,647][INFO][o.e.c.m.MetaDataDeleteIndexService] [gdph-es-01] [gh-partition-000539/Q5jRkFC5RDWt76DMzdFK\_g] deleting index [2018-12-15T16:30:04,790][DEBUG][o.e.a.a.c.s.TransportClusterStatsAction] [gdph-es-01] failed to execute on node [mGitnIAFQuilix-OjpWU8g] org.elasticsearch.transport.RemoteTransportException: [gdph-es-03][10.201.236.37:9201][cluster:monitor/stats[n]] Caused by: org.apache.lucene.store.AlreadyClosedException: engine is closed at org.elasticsearch.index.shard.IndexShard.getEngine(IndexShard.java:1988) ~[elasticsearch-6.3.2.jar:6.3.2] at org.elasticsearch.index.shard.IndexShard.segmentStats(IndexShard.java:961) ~[elasticsearch-6.3.2.jar:6.3.2] at org.elasticsearch.action.admin.indices.stats.CommonStats.(CommonStats.java:211) ~[elasticsearch-6.3.2.jar:6.3.2] at org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction.nodeOperation(TransportClusterStatsAction.java:106) ~[elasticsearch-6.3.2.jar:6.3.2] at org.elasticsearch.action.admin.cluster.stats.TransportClusterStatsAction.nodeOperation(TransportClusterStatsAction.java:50) ~[elasticsearch-6.3.2.jar:6.3.2] at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:140) ~[elasticsearch-6.3.2.jar:6.3.2] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:260) ~[elasticsearch-6.3.2.jar:6.3.2] at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:256) ~[elasticsearch-6.3.2.jar:6.3.2] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:259) ~[?:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.3.2.jar:6.3.2] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:317) ~[?:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.3.2.jar:6.3.2] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1592) ~[elasticsearch-6.3.2.jar:6.3.2] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:725) ~[elasticsearch-6.3.2.jar:6.3.2] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.3.2.jar:6.3.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0\_171] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0\_171] at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_171] Since this happens during curator action can it possibly be a race condition and if so any ideas how to avoid it? Client service itself seems to be unaffected,error rate and response times did not change. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 7, 2019, 4:24pm UTC](https://discuss.elastic.co/t/alreadyclosedexception-engine-is-closed-during-cluster-stats/163238/2 "2019-01-07T16:24:38Z")
This is a `DEBUG`-level log message, so does not indicate any particular problem that needs resolving. You are right that it indicates that an index was being deleted while its stats were being requested. ---
**Author:** ![katre](https://avatars.discourse-cdn.com/v4/letter/k/f6c823/32.png) [@katre](https://discuss.elastic.co/u/katre)\ **Post date:** [January 7, 2019, 4:29pm UTC](https://discuss.elastic.co/t/alreadyclosedexception-engine-is-closed-during-cluster-stats/163238/3 "2019-01-07T16:29:27Z")
Thank you, it means we need to rethink our monitoring then. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 7, 2019, 4:35pm UTC](https://discuss.elastic.co/t/alreadyclosedexception-engine-is-closed-during-cluster-stats/163238/4 "2019-01-07T16:35:04Z")
It looks like we log `DEBUG`-level logs for `org.elasticsearch.action` by default, because of this section in the `log4j2.properties` file in the config directory: ```auto # log action execution errors for easier debugging logger.action.name = org.elasticsearch.action logger.action.level = debug ``` If you change `debug` to `info` then you will not see these messages again. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 4, 2019, 4:35pm UTC](https://discuss.elastic.co/t/alreadyclosedexception-engine-is-closed-during-cluster-stats/163238/5 "2019-02-04T16:35:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.