OutOfMemory error


(Pranav) #1

Hi, I'm facing the issue that my Elasticsearch data nodes shutdown automatically for some searches.

Elasticsearch version - 6.3.2
Ram - 8gb
Heap size - 1gb
Documents - 7000(max)

This is the error ->

at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:48)
    at java.util.TreeMap.put(TreeMap.java:577)
       Local Variable: java.util.TreeMap#2
       Local Variable: java.lang.Integer#160
       Local Variable: java.util.TreeMap$Entry#149
       Local Variable: java.lang.Integer#294
    at org.apache.lucene.util.automaton.SortedIntSet.incr(SortedIntSet.java:53)
    at org.apache.lucene.util.automaton.Operations.determinize(Operations.java:780)
       Local Variable: org.apache.lucene.util.automaton.Automaton$Builder#2
       Local Variable: java.util.HashMap#3
       Local Variable: java.util.ArrayDeque#2
       Local Variable: org.apache.lucene.util.automaton.SortedIntSet#2
       Local Variable: org.apache.lucene.util.automaton.Operations$PointTransitionSet#2
       Local Variable: int[]#8065
       Local Variable: org.apache.lucene.util.automaton.Automaton#1980
    at org.apache.lucene.util.automaton.Operations.getCommonSuffixBytesRef(Operations.java:1155)
    at org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:238)
       Local Variable: org.apache.lucene.util.automaton.CompiledAutomaton#1108
       Local Variable: org.apache.lucene.util.automaton.Automaton#1979
    at org.apache.lucene.search.AutomatonQuery.<init>(AutomatonQuery.java:104)
    at org.apache.lucene.search.AutomatonQuery.<init>(AutomatonQuery.java:81)
    at org.apache.lucene.search.WildcardQuery.<init>(WildcardQuery.java:67)
    at org.apache.lucene.queryparser.classic.QueryParserBase.newWildcardQuery(QueryParserBase.java:644)
    at org.apache.lucene.queryparser.classic.QueryParserBase.getWildcardQuery(QueryParserBase.java:703)
    at org.elasticsearch.index.search.QueryStringQueryParser.getWildcardQuerySingle(QueryStringQueryParser.java:666)
       Local Variable: java.lang.String#170193
    at org.elasticsearch.index.search.QueryStringQueryParser.getWildcardQuery(QueryStringQueryParser.java:640)
       Local Variable: java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$1#2
       Local Variable: java.lang.String#220606
       Local Variable: java.util.ArrayList#9417
       Local Variable: java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$UnmodifiableEntry#4
    at org.apache.lucene.queryparser.classic.QueryParserBase.handleBareTokenQuery(QueryParserBase.java:820)
    at org.apache.lucene.queryparser.classic.QueryParser.Term(QueryParser.java:469)
    at org.apache.lucene.queryparser.classic.QueryParser.Clause(QueryParser.java:355)
    at org.apache.lucene.queryparser.classic.QueryParser.Query(QueryParser.java:244)
       Local Variable: java.util.ArrayList#9418
    at org.apache.lucene.queryparser.classic.QueryParser.TopLevelQuery(QueryParser.java:215)
    at org.apache.lucene.queryparser.classic.QueryParserBase.parse(QueryParserBase.java:109)
       Local Variable: java.lang.String#220898
       Local Variable: org.elasticsearch.index.search.QueryStringQueryParser#3
    at org.elasticsearch.index.search.QueryStringQueryParser.parse(QueryStringQueryParser.java:786)
    at org.elasticsearch.index.query.QueryStringQueryBuilder.doToQuery(QueryStringQueryBuilder.java:1041)
    at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:98)
       Local Variable: org.elasticsearch.index.query.QueryStringQueryBuilder#70
    at org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:404)
       Local Variable: java.util.ArrayList$Itr#3
    at org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:378)
       Local Variable: org.apache.lucene.search.BooleanQuery$Builder#3
    at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:98)
       Local Variable: org.elasticsearch.index.query.BoolQueryBuilder#54
    at org.elasticsearch.index.query.BoolQueryBuilder.addBooleanClauses(BoolQueryBuilder.java:404)
       Local Variable: java.util.ArrayList$Itr#4
    at org.elasticsearch.index.query.BoolQueryBuilder.doToQuery(BoolQueryBuilder.java:380)
       Local Variable: org.apache.lucene.search.BooleanQuery$Builder#4
    at org.elasticsearch.index.query.AbstractQueryBuilder.toQuery(AbstractQueryBuilder.java:98)
    at org.elasticsearch.index.query.QueryShardContext.lambda$toQuery$2(QueryShardContext.java:304)
    at org.elasticsearch.index.query.QueryShardContext$$Lambda$2900.apply(<unknown string>)
    at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:316)
       Local Variable: org.elasticsearch.index.query.QueryShardContext$$Lambda$2900#2
    at org.elasticsearch.index.query.QueryShardContext.toQuery(QueryShardContext.java:303)
       Local Variable: org.elasticsearch.index.query.BoolQueryBuilder#53
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:724)
       Local Variable: java.util.HashMap#63550
       Local Variable: org.elasticsearch.search.builder.SearchSourceBuilder#14
       Local Variable: org.elasticsearch.index.query.QueryShardContext#3
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:575)
       Local Variable: org.elasticsearch.search.DefaultSearchContext#2
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:551)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:347)
       Local Variable: org.elasticsearch.action.search.SearchTask#11
    at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:333)
    at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:329)
       Local Variable: org.elasticsearch.search.internal.ShardSearchTransportRequest#14
       Local Variable: org.elasticsearch.search.SearchService$2#11
    at org.elasticsearch.search.SearchService$3.doRun(SearchService.java:1019)
       Local Variable: org.elasticsearch.search.SearchService$3#11

On evaluating the ".hprof" file for heap dump these are the problem suspects:
Screenshot%20(40)

Can you guys tell me what is the problem here? Is it because of my queries or less heap size?


(David Pilato) #2

I'd first use up to half of the RAM available as the heap size. 4 gb here.


(Pranav) #3

Thanks @dadoonet
I understand Heap size could be one of the reasons,
Is there anything other than this that can cause out of memory issues?
Will having fields over 1000 in each index affect memory?


(David Pilato) #4

Yes.


(Pranav) #5

what can they be?


(David Pilato) #6

As you have a small number of documents, that's probably more related to the content, fields, query you are running....


(Pranav) #7

Thank you @dadoonet
Question 1: As I could see in a stacktrace, It is showing wildcard query.

at org.apache.lucene.search.AutomatonQuery.<init>(AutomatonQuery.java:104)
    at org.apache.lucene.search.AutomatonQuery.<init>(AutomatonQuery.java:81)
    at org.apache.lucene.search.WildcardQuery.<init>(WildcardQuery.java:67)
    at org.apache.lucene.queryparser.classic.QueryParserBase.newWildcardQuery(QueryParserBase.java:644)
    at org.apache.lucene.queryparser.classic.QueryParserBase.getWildcardQuery(QueryParserBase.java:703)
    at org.elasticsearch.index.search.QueryStringQueryParser.getWildcardQuerySingle(QueryStringQueryParser.java:666)

As a wildcard query internally does a lot of computation, So could it be the reason for Out of memory?

Question 2: The problem suspects which are shown by a heap dump analysis:

One instance of "org.elasticsearch.search.SearchService" loaded by "sun.misc.Launcher$AppClassLoader @ Oxca660000" occupies 484,600,608 (46.12%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Node[]" loaded by "<system class loader>". 
Keywords java.util.concurrent.ConcurrentHashMap$Nodep sun.misc.Launcher$AppClassLoader @ Oxca660000 org.elasticsearch.search.SearchService 

ConcurrentHashMap object is occupying my lot of memory. Can you tell when this object is created and used?


(David Pilato) #8

If you have a lot of terms I guess so.
What is your query ?

I can't.

But have a look at Elastic Mapping explosion


(Pranav) #9

Thanks for the reply
As I see in my code, I have a query running with wildcard

   {  "from" : 0, "size" : 5,"_source": {"includes": [ ]
    },  "query": {
        "bool": {"should": [   {
        "bool": {"must": [ {
              "query_string": {
                "query":"*only* *search* *for* *pranav*",
                "fields": []
    }}]}}],"minimum_should_match":1}},"highlight": {
            "type" : "unified",
               "fragment_size" : 0,
               "order":"score",
    			"number_of_fragments" :4,
            "fields": {
                "*": {
                }           	 
            }
        }}  

This query continuously hits my Elastic server when a new character is typed. Something like AutoSuggestion but not exactly the same. Can this query be the reason fro my OutOfMemory issue?


(David Pilato) #10

Probably. You are using wildcard queries which not recommended.
Anyway, as we saw you have plenty of RAM available, just upgrade the HEAP to 4 gb and you should be good.


(Bernt Rostad) #11

I've seen a single query, with an asterisk prefix, eat all 29 G heap size on a node, causing it to crash with OutOfMemory, so yes - wildcards can cause OutOfMemory errors. Even if you add more Java Heap Size.

I have tried to play with the circuit breaker settings, to terminate such dangerous queries, but without luck so far. So your best line of action is probably to prevent such queries from being run in the first place but also have a good system for detecting node crashes and restarting them to avoid a red cluster state.


(Pranav) #12

Thank you so much @dadoonet @Bernt_Rostad for your response.

I have increased the HEAP size and didn't get any issue from past 2 days. I am also planning to reduce the use cases of a wild card.

Just for curiosity, I want to know something about Mapping Explosion.
As I have about 1800 fields, I know this number is too large. As Elasticsearch provides functionality to increase the number of fields. So by increasing it to a large value can cause any issues?


(system) closed #13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.