# Elasticsearch data node out of memory **URL:** \ **Category:** Elasticsearch\ **Created:** [February 17, 2023, 7:09pm UTC](https://discuss.elastic.co/t/elasticsearch-data-node-out-of-memory/325866 "2023-02-17T19:09:51Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![sssamant](https://avatars.discourse-cdn.com/v4/letter/s/eada6e/32.png) [@sssamant](https://discuss.elastic.co/u/sssamant)\ **Post date:** [February 17, 2023, 7:09pm UTC](https://discuss.elastic.co/t/elasticsearch-data-node-out-of-memory/325866/1 "2023-02-17T19:09:51Z")
Hello everyone, We are having out of memory issue for the elasticsearch data nodes? Can you please help me out to find the issue. Here is log from elasticsearch cluster. [2023-02-17 10:02:47,551][WARN][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the selector loop. at org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:213) at org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:187) at org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:95) at org.elasticsearch.search.SearchService.parseSource(SearchService.java:838) ... 12 more [2023-02-17 06:35:35,197][DEBUG][action.search] [dn8] All shards failed for phase: [query] : {"excludes": , "includes": ["ingdt"]}, "from": 0, "size": 1}]]; nested: SearchParseException[No mapping found for [ingdt] in order to sort on];[{"sort": {"ingdt": {"order": "desc"}}, "query": {"boolseException[No mapping found for [ingdt] in order to sort on];[{"sort": {"ingdt": {"order": "desc"}}, "query"at org.elasticsearch.search.SearchService.parseSource(SearchService.java:855)es": ["ingdt"]}, "from":at org.elasticsearch.search.SearchService.createContext(SearchService.java:654) at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:620) at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:371) at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReat org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReat org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:3at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.jat org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandlerat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: SearchParseException[No mapping found for [ingdt] in order to sort on] at org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:187) ... 12 moreticsearch.search.SearchService.parseSource(SearchService.java:838):95) [2023-02-17 10:00:11,868][INFO][monitor.jvm] [dn8] [gc][old][157114][139] duration [6.9s], collections [1]/[7.4s], total [6.9s]/[6.3m], memory [12.4gb]-\>[9gb]/[31.8gb], all\_pools {[young]8.1mb]-\>[0b]/[108.1mb]}{[old] [12.3gb]-\>[8.9gb]/[30.9gb]} [2023-02-17 10:00:28,157][INFO][monitor.jvm] [dn8] [gc][old][157121][140] duration [9.2s], collections [1]/[9.9s], total [9.2s]/[6.4m], memory [28.9gb]-\>[31.1gb]/[31.8gb], all\_pools {[you] [108.1mb]-\>[0b]/[108.1mb]}{[old] [28.6gb]-\>[30.9gb]/[30.9gb]} [2023-02-17 10:00:36,278][INFO][monitor.jvm] [dn8] [gc][old][157122][141] duration [8s], collections [1]/[8.1s], total [8s]/[6.6m], memory [31.1gb]-\>[31.6gb]/[31.8gb], all\_pools {[young] b]-\>[0b]/[108.1mb]}{[old] [30.9gb]-\>[30.9gb]/[30.9gb]} [2023-02-17 10:00:44,582][INFO][monitor.jvm] [dn8] [gc][old][157123][142] duration [8.2s], collections [1]/[8.3s], total [8.2s]/[6.7m], memory [31.6gb]-\>[31.8gb]/[31.8gb], all\_pools {[youor] [0b]-\>[69.5mb]/[108.1mb]}{[old] [30.9gb]-\>[30.9gb]/[30.9gb]} [2023-02-17 10:02:47,551][WARN][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the selector loop. java.lang.OutOfMemoryError: Java heap space [2023-02-17 10:04:13,535][WARN][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the selector loop. java.lang.OutOfMemoryError: Java heap space [2023-02-17 10:02:54,144][WARN][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the selector loop. java.lang.OutOfMemoryError: Java heap space [2023-02-17 10:02:47,551][WARN][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the selector loop. .2s], collections [1]/[9.9s], total [9.2s]/[6.4m], memory [28.9gb]-\>[31.1gb]/[31.8gb], all\_pools {[young] [187.3mb]-\>[177mb]/[865.3mb]}{[survivor] [108.1mb]-\>[0b]/[108.1mb]}{[old] [28.6gb]-\>[30.9gb]/[30.9gb]} [2023-02-17 10:00:36,278][INFO][monitor.jvm] [dn8] [gc][old][157122][141] duration [8s], collections [1]/[8.1s], total [8s]/[6.6m], memory [31.1gb]-\>[31.6gb]/[31.8gb], all\_pools {[young] [177mb]-\>[740.2mb]/[865.3mb]}{[survivor] [0b]-\>[0b]/[108.1mb]}{[old] [30.9gb]-\>[30.9gb]/[30.9gb]} [2023-02-17 10:00:44,582][INFO][monitor.jvm] [dn8] [gc][old][157123][142] duration [8.2s], collections [1]/[8.3s], total [8.2s]/[6.7m], memory [31.6gb]-\>[31.8gb]/[31.8gb], all\_pools {[young] [740.2mb]-\>[865.3mb]/[865.3mb]}{[survivor] [0b]-\>[69.5mb]/[108.1mb]}{[old] [30.9gb]-\>[30.9gb]/[30.9gb]} [2023-02-17 10:02:47,551][WARN][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the selector loop. java.lang.OutOfMemoryError: Java heap space [2023-02-17 10:04:13,535][WARN][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the selector loop. java.lang.OutOfMemoryError: Java heap space [2023-02-17 10:02:54,144][WARN][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the selector loop. java.lang.OutOfMemoryError: Java heap space [2023-02-17 10:02:47,551][WARN][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the selector loop. java.lang.OutOfMemoryError: Java heap space ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 17, 2023, 7:18pm UTC](https://discuss.elastic.co/t/elasticsearch-data-node-out-of-memory/325866/2 "2023-02-17T19:18:58Z")
Which version of Elasticsearch are you using? What is the full output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/8.6/cluster-stats.html)? What is the size and hardware specification of the cluster? ---
**Author:** ![sssamant](https://avatars.discourse-cdn.com/v4/letter/s/eada6e/32.png) [@sssamant](https://discuss.elastic.co/u/sssamant)\ **Post date:** [February 17, 2023, 7:48pm UTC](https://discuss.elastic.co/t/elasticsearch-data-node-out-of-memory/325866/3 "2023-02-17T19:48:36Z")
Cluster and version: { "timestamp" : 1676663158235, "cluster\_name" : "psw", "status" : "green", "indices" : { "count" : 103, "shards" : { "total" : 1100, "primaries" : 550, "replication" : 1.0, "index" : { "shards" : { "min" : 5, "max" : 80, "avg" : 10.679611650485437 }, "primaries" : { "min" : 5, "max" : 40, "avg" : 5.339805825242719 }, "replication" : { "min" : 0.0, "max" : 2.0, "avg" : 1.0 } } }, "docs" : { "count" : 796821702, "deleted" : 55324205 }, "store" : { "size\_in\_bytes" : 1798260205580, "throttle\_time\_in\_millis" : 0 }, "fielddata" : { "memory\_size\_in\_bytes" : 16909107400, "evictions" : 0 }, "query\_cache" : { "memory\_size\_in\_bytes" : 1685706352, "total\_count" : 179648382, "hit\_count" : 7978360, "miss\_count" : 171670022, "cache\_size" : 61342, "cache\_count" : 63404, "evictions" : 2062 }, "completion" : { "size\_in\_bytes" : 0 }, "segments" : { "count" : 7169, "memory\_in\_bytes" : 3236880780, "terms\_memory\_in\_bytes" : 2728959888, "stored\_fields\_memory\_in\_bytes" : 454045704, "term\_vectors\_memory\_in\_bytes" : 0, "norms\_memory\_in\_bytes" : 36443264, "doc\_values\_memory\_in\_bytes" : 17431924, "index\_writer\_memory\_in\_bytes" : 0, "index\_writer\_max\_memory\_in\_bytes" : 67071705088, "version\_map\_memory\_in\_bytes" : 0, "fixed\_bit\_set\_memory\_in\_bytes" : 3066088 }, "percolate" : { "total" : 0, "time\_in\_millis" : 0, "current" : 0, "memory\_size\_in\_bytes" : -1, "memory\_size" : "-1b", "queries" : 0 } }, "nodes" : { "count" : { "total" : 9, "master\_only" : 0, "data\_only" : 6, "master\_data" : 3, "client" : 0 }, "versions" : ["2.3.5"], "os" : { "available\_processors" : 144, "allocated\_processors" : 144, "mem" : { "total\_in\_bytes" : 10750435328 }, "names" : [ { "name" : "Linux", "count" : 9 } ] }, "process" : { "cpu" : { "percent" : 0 }, "open\_file\_descriptors" : { "min" : 1788, "max" : 2021, "avg" : 1891 } }, "jvm" : { "max\_uptime\_in\_millis" : 31383265, "versions" : [ { "version" : "1.8.0\_181", "vm\_name" : "OpenJDK 64-Bit Server VM", "vm\_version" : "25.181-b13", "vm\_vendor" : "Oracle Corporation", "count" : 7 }, { "version" : "1.8.0\_242", "vm\_name" : "OpenJDK 64-Bit Server VM", "vm\_version" : "25.242-b08", "vm\_vendor" : "Oracle Corporation", "count" : 2 } ], "mem" : { "heap\_used\_in\_bytes" : 90600761640, "heap\_max\_in\_bytes" : 308217249792 }, "threads" : 1662 }, "fs" : { "total\_in\_bytes" : 14266797133824, "free\_in\_bytes" : 12238875586560, "available\_in\_bytes" : 11637248139264 }, "plugins" : [ { "name" : "head", "version" : "master", "description" : "head - A web front end for an Elasticsearch cluster", "url" : "/\_plugin/head/", "jvm" : false, "site" : true }, { "name" : "cloud-aws", "version" : "2.3.5", "description" : "The Amazon Web Service (AWS) Cloud plugin allows to use AWS API for the unicast discovery mechanism and add S3 repositories.", "jvm" : true, "classname" : "org.elasticsearch.plugin.cloud.aws.CloudAwsPlugin", "isolated" : true, "site" : false }, { "name" : "delete-by-query", "version" : "2.3.5", "description" : "The Delete By Query plugin allows to delete documents in Elasticsearch with a single query.", "jvm" : true, "classname" : "org.elasticsearch.plugin.deletebyquery.DeleteByQueryPlugin", "isolated" : true, "site" : false }, { "name" : "sql", "version" : "2.3.5.0", "description" : "Query elasticsearch using SQL", "url" : "/\_plugin/sql/", "jvm" : true, "classname" : "org.elasticsearch.plugin.nlpcn.SqlPlug", "isolated" : true, "site" : true } ] } } Hardaware: 8X64( 8 cpu's and 64 ram) Assinged heap size for data node is 32gb(50%) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 17, 2023, 8:27pm UTC](https://discuss.elastic.co/t/elasticsearch-data-node-out-of-memory/325866/4 "2023-02-17T20:27:18Z")
> [@sssamant](#): > > "versions" : ["2.3.5"], That is a very, very old version that has been EOL a long time. I have not used that version in many years so do not think I will be able to help much. I would recommend upgrading. > [@sssamant](#): > > { > "name" : "sql", > "version" : "2.3.5.0", > "description" : "Query elasticsearch using SQL", > "url" : "/\_plugin/sql/", > "jvm" : true, > "classname" : "org.elasticsearch.plugin.nlpcn.SqlPlug", > "isolated" : true, > "site" : true > } It also seems like you are using a third-party plugin that I have never used and that may very well contribute to heap usage. ---
**Author:** ![sssamant](https://avatars.discourse-cdn.com/v4/letter/s/eada6e/32.png) [@sssamant](https://discuss.elastic.co/u/sssamant)\ **Post date:** [February 17, 2023, 9:33pm UTC](https://discuss.elastic.co/t/elasticsearch-data-node-out-of-memory/325866/5 "2023-02-17T21:33:59Z")
Ok, thank you for your help. ---
**Author:** ![sssamant](https://avatars.discourse-cdn.com/v4/letter/s/eada6e/32.png) [@sssamant](https://discuss.elastic.co/u/sssamant)\ **Post date:** [February 17, 2023, 9:46pm UTC](https://discuss.elastic.co/t/elasticsearch-data-node-out-of-memory/325866/6 "2023-02-17T21:46:54Z")
I am suspecting that they are running search query against filed ingdt for sorting through out the whole index, that is causing the out of memory issue. See the snippet of log. Are you agree with me? All shards failed for phase: [query] : {"excludes": , "includes": ["ingdt"]}, "from": 0, "size": 1}]]; nested: SearchParseException[No mapping found for [ingdt] in order to sort on];[{"sort": {"ingdt": {"order": "desc"}}, "query": {"boolseException[No mapping found for [ingdt] in order to sort on];[{"sort": {"ingdt": {"order": "desc"}}, "query"at org.elasticsearch.search.SearchService.parseSource(SearchService.java:855)es": ["ingdt"]}, "from":at org.elasticsearch.search.SearchService.createContext(SearchService.java:654) at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:620) at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:371) ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 20, 2023, 2:26am UTC](https://discuss.elastic.co/t/elasticsearch-data-node-out-of-memory/325866/7 "2023-02-20T02:26:02Z")
2.X is 7 years old, which means there's likely not a lot of retained memory on debugging this version given we are up to 8.6, and you are unlikely to get much advice other than what Christian mentioned - **upgrade ASAP**. > [@sssamant](#): > > I am suspecting that they are running search query against filed ingdt for sorting through out the whole index, that is causing the out of memory issue. See the snippet of log. Are you agree with me? Possibly? Again, I don't remember most of what 2.4 did to comment with any assurance. Also one final comment - please format your code/logs/config using the `` button, or markdown style back ticks. It helps to make things easy to read which helps us help you 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 20, 2023, 2:26am UTC](https://discuss.elastic.co/t/elasticsearch-data-node-out-of-memory/325866/8 "2023-03-20T02:26:56Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.