OutOfMemory error

pranav24 · January 2, 2019, 8:22am

Hi, I'm facing the issue that my Elasticsearch data nodes shutdown automatically for some searches.

Elasticsearch version - 6.3.2
Ram - 8gb
Heap size - 1gb
Documents - 7000(max)

This is the error ->

at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:48)
    at java.util.TreeMap.put(TreeMap.java:577)
       Local Variable: java.util.TreeMap#2
       Local Variable: java.lang.Integer#160
       Local Variable: java.util.TreeMap$Entry#149
       Local Variable: java.lang.Integer#294
    at org.apache.lucene.util.automaton.SortedIntSet.incr(SortedIntSet.java:53)
    at org.apache.lucene.util.automaton.Operations.determinize(Operations.java:780)
       Local Variable: org.apache.lucene.util.automaton.Automaton$Builder#2
       Local Variable: java.util.HashMap#3
       Local Variable: java.util.ArrayDeque#2
       Local Variable: org.apache.lucene.util.automaton.SortedIntSet#2
       Local Variable: org.apache.lucene.util.automaton.Operations$PointTransitionSet#2
       Local Variable: int[]#8065
       Local Variable: org.apache.lucene.util.automaton.Automaton#1980
    at org.apache.lucene.util.automaton.Operations.getCommonSuffixBytesRef(Operations.java:1155)
    at org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:238)
       Local Variable: org.apache.lucene.util.automaton.CompiledAutomaton#1108
       Local Variable: org.apache.lucene.util.automaton.Automaton#1979
    at org.apache.lucene.search.AutomatonQuery.<init>(AutomatonQuery.java:104)
    at org.apache.lucene.search.AutomatonQuery.<init>(AutomatonQuery.java:81)
    at org.apache.lucene.search.WildcardQuery.<init>(WildcardQuery.java:67)
    at org.apache.lucene.queryparser.classic.QueryParserBase.newWildcardQuery(QueryParserBase.java:644)
    at org.apache.lucene.queryparser.classic.QueryParserBase.getWildcardQuery(QueryParserBase.java:703)
    at org.elasticsearch.index.search.QueryStringQueryParser.getWildcardQuerySingle(QueryStringQueryParser.java:666)
       Local Variable: java.lang.String#170193
    at org.elasticsearch.index.search.QueryStringQueryParser.getWildcardQuery(QueryStringQueryParser.java:640)
       Local Variable: java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$1#2
       Local Variable: java.lang.String#220606
       Local Variable: java.util.ArrayList#9417
       Local Variable: java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$UnmodifiableEntry#4
    at org.apache.lucene.queryparser.classic.QueryParserBase.handleBareTokenQuery(QueryParserBase.java:820)
    at org.apache.lucene.queryparser.classic.QueryParser.Term(QueryParser.java:469)
    at org.apache.lucene.queryparser.classic.QueryParser.Clause(QueryParser.java:355)
    at org.apache.lucene.queryparser.classic.QueryParser.Query(QueryParser.java:244)
       Local Variable: java.util.ArrayList#9418
    at org.apache.lucene.queryparser.classic.QueryParser.TopLevelQuery(QueryParser.java:215)
    at org.apache.lucene.queryparser.classic.QueryParserBase.parse(QueryParserBase.java:109)
       Local Variable: java.lang.String#220898
       Local Variable: org.elasticsearch.index.search.QueryStringQueryParser#3
    at org.elasticsearch.index.search.QueryStringQueryParser.parse(QueryStringQueryParser.java:786)
    at org.elasticsearch.index.query.QueryStringQueryBuilder.doToQuery(QueryStringQueryBuilder.java:1041)
    at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:98)
       Local Variable: org.elasticsearch.index.query.QueryStringQueryBuilder#70
    at org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:404)
       Local Variable: java.util.ArrayList$Itr#3
    at org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:378)
       Local Variable: org.apache.lucene.search.BooleanQuery$Builder#3
    at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:98)
       Local Variable: org.elasticsearch.index.query.BoolQueryBuilder#54
    at org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:404)
       Local Variable: java.util.ArrayList$Itr#4
    at org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:380)
       Local Variable: org.apache.lucene.search.BooleanQuery$Builder#4
    at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:98)
    at org.elasticsearch.index.query.QueryShardContext.lambda$toQuery$2(QueryShardContext.java:304)
    at org.elasticsearch.index.query.QueryShardContext$$Lambda$2900.apply(<unknown string>)
    at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:316)
       Local Variable: org.elasticsearch.index.query.QueryShardContext$$Lambda$2900#2
    at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:303)
       Local Variable: org.elasticsearch.index.query.BoolQueryBuilder#53
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:724)
       Local Variable: java.util.HashMap#63550
       Local Variable: org.elasticsearch.search.builder.SearchSourceBuilder#14
       Local Variable: org.elasticsearch.index.query.QueryShardContext#3
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:575)
       Local Variable: org.elasticsearch.search.DefaultSearchContext#2
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:551)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:347)
       Local Variable: org.elasticsearch.action.search.SearchTask#11
    at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:333)
    at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:329)
       Local Variable: org.elasticsearch.search.internal.ShardSearchTransportRequest#14
       Local Variable: org.elasticsearch.search.SearchService$2#11
    at org.elasticsearch.search.SearchService$3.doRun(SearchService.java:1019)
       Local Variable: org.elasticsearch.search.SearchService$3#11

On evaluating the ".hprof" file for heap dump these are the problem suspects:
Screenshot%20(40)

Can you guys tell me what is the problem here? Is it because of my queries or less heap size?

dadoonet · January 2, 2019, 11:38am

I'd first use up to half of the RAM available as the heap size. 4 gb here.

pranav24 · January 2, 2019, 11:43am

Thanks @dadoonet
I understand Heap size could be one of the reasons,
Is there anything other than this that can cause out of memory issues?
Will having fields over 1000 in each index affect memory?

dadoonet · January 2, 2019, 2:51pm

Yes.

pranav24 · January 3, 2019, 6:06am

what can they be?

dadoonet · January 3, 2019, 6:26am

As you have a small number of documents, that's probably more related to the content, fields, query you are running....

pranav24 · January 3, 2019, 6:39am

Thank you @dadoonet
Question 1: As I could see in a stacktrace, It is showing wildcard query.

at org.apache.lucene.search.AutomatonQuery.<init>(AutomatonQuery.java:104)
    at org.apache.lucene.search.AutomatonQuery.<init>(AutomatonQuery.java:81)
    at org.apache.lucene.search.WildcardQuery.<init>(WildcardQuery.java:67)
    at org.apache.lucene.queryparser.classic.QueryParserBase.newWildcardQuery(QueryParserBase.java:644)
    at org.apache.lucene.queryparser.classic.QueryParserBase.getWildcardQuery(QueryParserBase.java:703)
    at org.elasticsearch.index.search.QueryStringQueryParser.getWildcardQuerySingle(QueryStringQueryParser.java:666)

As a wildcard query internally does a lot of computation, So could it be the reason for Out of memory?

Question 2: The problem suspects which are shown by a heap dump analysis:

One instance of "org.elasticsearch.search.SearchService" loaded by "sun.misc.Launcher$AppClassLoader @ Oxca660000" occupies 484,600,608 (46.12%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Node[]" loaded by "<system class loader>". 
Keywords java.util.concurrent.ConcurrentHashMap$Nodep sun.misc.Launcher$AppClassLoader @ Oxca660000 org.elasticsearch.search.SearchService

ConcurrentHashMap object is occupying my lot of memory. Can you tell when this object is created and used?

dadoonet · January 3, 2019, 7:45am

If you have a lot of terms I guess so.
What is your query ?

I can't.

But have a look at Elastic Mapping explosion - #7 by Mark_Harwood

pranav24 · January 3, 2019, 8:16am

Thanks for the reply
As I see in my code, I have a query running with wildcard

   {  "from" : 0, "size" : 5,"_source": {"includes": [ ]
    },  "query": {
        "bool": {"should": [   {
        "bool": {"must": [ {
              "query_string": {
                "query":"*only* *search* *for* *pranav*",
                "fields": []
    }}]}}],"minimum_should_match":1}},"highlight": {
            "type" : "unified",
               "fragment_size" : 0,
               "order":"score",
    			"number_of_fragments" :4,
            "fields": {
                "*": {
                }           	 
            }
        }}

This query continuously hits my Elastic server when a new character is typed. Something like AutoSuggestion but not exactly the same. Can this query be the reason fro my OutOfMemory issue?

dadoonet · January 3, 2019, 8:37am

Probably. You are using wildcard queries which not recommended.
Anyway, as we saw you have plenty of RAM available, just upgrade the HEAP to 4 gb and you should be good.

Bernt_Rostad · January 3, 2019, 9:51am

I've seen a single query, with an asterisk prefix, eat all 29 G heap size on a node, causing it to crash with OutOfMemory, so yes - wildcards can cause OutOfMemory errors. Even if you add more Java Heap Size.

I have tried to play with the circuit breaker settings, to terminate such dangerous queries, but without luck so far. So your best line of action is probably to prevent such queries from being run in the first place but also have a good system for detecting node crashes and restarting them to avoid a red cluster state.

pranav24 · January 3, 2019, 10:25am

Thank you so much @dadoonet @Bernt_Rostad for your response.

I have increased the HEAP size and didn't get any issue from past 2 days. I am also planning to reduce the use cases of a wild card.

Just for curiosity, I want to know something about Mapping Explosion.
As I have about 1800 fields, I know this number is too large. As Elasticsearch provides functionality to increase the number of fields. So by increasing it to a large value can cause any issues?

system · January 31, 2019, 10:25am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Best way to resolve this out of memory error in my ES Data Nodes Elasticsearch	3	880	March 6, 2021
Elastic Search-Java.lang.OutOfMemoryError Elasticsearch	2	1804	October 15, 2018
java.lang.OutOfMemoryError at org.apache.lucene.util.packed.Direct8.<init>(Direct8.java:36) Elasticsearch	1	653	February 9, 2018
Elasticsearch (6.4.1) - JVM OutOfMemoryError Elasticsearch	5	1008	June 26, 2019
OutOfMemoryError Java Heap Space Elasticsearch	3	874	May 24, 2017

OutOfMemory error

Related topics