# "failed to execute on node" Exception on elasticsearch 2.3.2 **URL:** \ **Category:** Elasticsearch\ **Created:** [May 10, 2016, 8:48am UTC](https://discuss.elastic.co/t/failed-to-execute-on-node-exception-on-elasticsearch-2-3-2/49623 "2016-05-10T08:48:33Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![macymin](https://avatars.discourse-cdn.com/v4/letter/m/8e7dd6/32.png) [@macymin](https://discuss.elastic.co/u/macymin)\ **Post date:** [May 10, 2016, 8:48am UTC](https://discuss.elastic.co/t/failed-to-execute-on-node-exception-on-elasticsearch-2-3-2/49623/1 "2016-05-10T08:48:33Z")
hi I have below exception consistently for all my three nodes' elasticsearch log file. The health status is green, can anyone help?? I think it causes the elasticsearch down after sometime as I just experienced one today (5/10) since last friday (5/6)'s cluster upgrade. [2016-05-10 16:12:39,369][DEBUG][action.admin.cluster.node.stats] [fslelkprod01] failed to execute on node [QSXAvrCzQQGDoprePsPzTQ] RemoteTransportException[[fslelkprod01][fslelkprod01/10.193.91.25:9300][cluster:monitor/nodes/stats[n]]]; nested: AlreadyClosedException[this IndexReader is closed]; Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexReader is closed at org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:274) at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:101) at org.apache.lucene.index.CompositeReader.getContext(CompositeReader.java:55) at org.apache.lucene.index.IndexReader.leaves(IndexReader.java:438) at org.elasticsearch.search.suggest.completion.Completion090PostingsFormat.completionStats(Completion090PostingsFormat.java:330) at org.elasticsearch.index.shard.IndexShard.completionStats(IndexShard.java:765) at org.elasticsearch.action.admin.indices.stats.CommonStats.(CommonStats.java:164) at org.elasticsearch.indices.IndicesService.stats(IndicesService.java:253) at org.elasticsearch.node.service.NodeService.stats(NodeService.java:158) at org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:82) at org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:44) at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:92) at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:230) at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:226) at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75) at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [2016-05-10 16:12:39,371][ERROR][marvel.agent.collector.node] [fslelkprod01] collector [node-stats-collector] - failed collecting data java.lang.ArrayIndexOutOfBoundsException: 0 at org.elasticsearch.action.support.nodes.BaseNodesResponse.getAt(BaseNodesResponse.java:72) at org.elasticsearch.marvel.agent.collector.node.NodeStatsCollector.doCollect(NodeStatsCollector.java:88) at org.elasticsearch.marvel.agent.collector.AbstractCollector.collect(AbstractCollector.java:99) at org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:187) at java.lang.Thread.run(Thread.java:745) ---
**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\ **Post date:** [May 10, 2016, 3:30pm UTC](https://discuss.elastic.co/t/failed-to-execute-on-node-exception-on-elasticsearch-2-3-2/49623/3 "2016-05-10T15:30:26Z")
Unfortunately, this was a bug introduced in 2.3.0. It was just fixed and will be released with 2.3.2: [https://github.com/elastic/elasticsearch/pull/18094](https://github.com/elastic/elasticsearch/pull/18094) ---
**Author:** ![macymin](https://avatars.discourse-cdn.com/v4/letter/m/8e7dd6/32.png) [@macymin](https://discuss.elastic.co/u/macymin)\ **Post date:** [May 11, 2016, 2:03am UTC](https://discuss.elastic.co/t/failed-to-execute-on-node-exception-on-elasticsearch-2-3-2/49623/4 "2016-05-11T02:03:46Z")
hi Zachary, thanks for the information. so how can we work around this before the 2.3.3 with bug fix is released? ---
**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\ **Post date:** [May 11, 2016, 2:47am UTC](https://discuss.elastic.co/t/failed-to-execute-on-node-exception-on-elasticsearch-2-3-2/49623/5 "2016-05-11T02:47:23Z")
Is your JVM crashing? Does it throw this exception right before crash? Unless you fall into the edge case with mmapfs crashing, this is basically a harmless exception. You'll see it spammed in your log a lot, but it's otherwise not a problem (the completion stats will be computed incorrectly, that's all). ---
**Author:** ![macymin](https://avatars.discourse-cdn.com/v4/letter/m/8e7dd6/32.png) [@macymin](https://discuss.elastic.co/u/macymin)\ **Post date:** [May 11, 2016, 3:18am UTC](https://discuss.elastic.co/t/failed-to-execute-on-node-exception-on-elasticsearch-2-3-2/49623/6 "2016-05-11T03:18:20Z")
i tried to search hs\_err\_pidXXXX.log, didnt see it. does it mean JVM not crash? ---
**Author:** ![bblank](https://avatars.discourse-cdn.com/v4/letter/b/f14d63/32.png) [@bblank](https://discuss.elastic.co/u/bblank)\ **Post date:** [August 31, 2016, 8:27pm UTC](https://discuss.elastic.co/t/failed-to-execute-on-node-exception-on-elasticsearch-2-3-2/49623/7 "2016-08-31T20:27:29Z")
Looks like I am hitting the same thing with 2.3.1. My indexing rates are really low compared to normal. My cluster log file only has a few entries... It doesn't appear to be a harmless event to our cluster. Is there anything else I can check? ---
**Author:** ![MarkOStewart](https://avatars.discourse-cdn.com/v4/letter/m/898d66/32.png) [@MarkOStewart](https://discuss.elastic.co/u/MarkOStewart)\ **Post date:** [September 8, 2016, 12:20am UTC](https://discuss.elastic.co/t/failed-to-execute-on-node-exception-on-elasticsearch-2-3-2/49623/8 "2016-09-08T00:20:49Z")
I would start a new thread as this one is a few months old and seems to have lived its useful life. That being said have you set the swappiness and vm.max\_max\_count. This made one of our clusters much happier on resources and stopped the swapping and helped i/o and index much faster. Hopefully you are on Linux? Another reason to start new thread you can give info about your setup. run sysctl vm.max\_map\_count it should return 262144 and cat /proc/sys/vm/swappiness should return 1 or 0 sysctl -w vm.max\_map\_count=262144 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:21pm UTC](https://discuss.elastic.co/t/failed-to-execute-on-node-exception-on-elasticsearch-2-3-2/49623/9 "2017-07-05T22:21:56Z")