java.lang.OutOfMemoryError: Java heap space - Bulk Indexing

Elastic Stack Elasticsearch
1

Single Node ES Cluster
ES version 2.3.2
ES_HEAP_SIZE=6G

From our application using multi curl api we are sending parallel bulk request to ES. Size of each bulk request is 10-15MB. Parallel requests send to ES are 100 requests( using multi curl api ). So we have 100 bulk requests each of sixe 10 MB approximately hitting ES. However I see java.lang.OutOfMemoryError: Java heap space.

java.lang.OutOfMemoryError: Java heap space
at com.fasterxml.jackson.core.util.TextBuffer.carr(TextBuffer.java:739)
at com.fasterxml.jackson.core.util.TextBuffer.resultArray(TextBuffer.java:726)
at com.fasterxml.jackson.core.util.TextBuffer.contentsAsArray(TextBuffer.java:367)
at com.fasterxml.jackson.core.util.TextBuffer.getTextBuffer(TextBuffer.java:316)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getTextCharacters(UTF8StreamJsonParser.java:415)
at com.fasterxml.jackson.core.JsonGenerator.copyCurrentEvent(JsonGenerator.java:1472)
at com.fasterxml.jackson.core.JsonGenerator.copyCurrentStructure(JsonGenerator.java:1578)
at com.fasterxml.jackson.core.JsonGenerator.copyCurrentStructure(JsonGenerator.java:1566)
at org.elasticsearch.common.xcontent.json.JsonXContentGenerator.copyCurrentStructure(JsonXContentGenerator.java:425)
at org.elasticsearch.common.xcontent.XContentBuilder.copyCurrentStructure(XContentBuilder.java:1180)
at org.elasticsearch.action.update.UpdateRequest.source(UpdateRequest.java:669)
at org.elasticsearch.action.bulk.BulkRequest.add(BulkRequest.java:399)
at org.elasticsearch.rest.action.bulk.RestBulkAction.handleRequest(RestBulkAction.java:88)
at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:54)
at org.elasticsearch.rest.RestController.executeHandler(RestController.java:205)

My question is

  1. Will increasing the indices.memory.index_buffer_size from 10% to bigger number would help to solve this issue
  2. How to find out how much memory is being used for bulk indexing Any stats api available?
  3. Do i need to change any other settings
2

gc stats collected during indexing

mem: {
heap_used: "4.1gb",
heap_used_in_bytes: 4471224576,
heap_used_percent: 71,
heap_committed: "5.8gb",
heap_committed_in_bytes: 6241845248,
heap_max: "5.8gb",
heap_max_in_bytes: 6241845248,
non_heap_used: "113.4mb",
non_heap_used_in_bytes: 118958944,
non_heap_committed: "116.1mb",
non_heap_committed_in_bytes: 121843712,
pools: {
young: {
used: "1.3gb",
used_in_bytes: 1480910984,
max: "1.4gb",
max_in_bytes: 1605304320,
peak_used: "1.4gb",
peak_used_in_bytes: 1605304320,
peak_max: "1.4gb",
peak_max_in_bytes: 1605304320
},
survivor: {
used: "41.4mb",
used_in_bytes: 43463840,
max: "191.3mb",
max_in_bytes: 200605696,
peak_used: "191.3mb",
peak_used_in_bytes: 200605696,
peak_max: "191.3mb",
peak_max_in_bytes: 200605696
},
old: {
used: "2.7gb",
used_in_bytes: 2946849752,
max: "4.1gb",
max_in_bytes: 4435935232,
peak_used: "3.9gb",
peak_used_in_bytes: 4241356112,
peak_max: "4.1gb",
peak_max_in_bytes: 4435935232
}
}
},
threads: {
count: 317,
peak_count: 321
},
gc: {
collectors: {
young: {
collection_count: 5309,
collection_time: "3.5m",
collection_time_in_millis: 210178
},
old: {
collection_count: 98,
collection_time: "6.7s",
collection_time_in_millis: 6783
}
}
}

3

hi,

1°) It may solve the issue, but it is only delaying the problem.
2°) I don't know if there is some stats like that
3°) I know the "threadpool.bulk.queue_size" setting which can be tweaked but I'm not sure if it could solve the problem..

Have you considered the possibilité of modifying the way ES is filled?

there are some possibilities I think in order to bound the number of parallel bulk ops to a reasonable limit.
Doing that, you will keep an availability of your node to perform some requests with a good perf, and you never have to hope that there will be enough memory..

what about:

  • reducing the number of parallel bulk requests?
  • keeping the same number of parallel bulk request but with less bulk operations inside?
  • adding a little "proxy" between your program and ES in order to spread the bulk requests to a reasonable cadence?

Otherwise, maybe you'll need to upgrade your cluster configuration by adding nodes / RAM

4

I had read in the document that when ES cannot handle bulk requests it would send rejections. But in my case, i don't see any bulk rejection rather OOM issue.

can you please explain a bit more on the above two statement.

I am trying this now. What can we do to achieve high ingestion rate?

A general question, From the _nodes/stats api , if i divide index total/ index time in milliseconds will i get ingestion rate that's doc per second.

5

@Tetrapack

Increasing the bulk queue pool would consume more memory right as now we are queuing more requests. how would it avoid OOM issue?

6

You're right, il you increase queue, it will consume more heap.. and fataly, not avoid OOM ^^'

When I talk about proxy, I meant, instead of sending requests to ES, send requests to a program which will buffer objects to index, then you can adjust more easily the sending rate.

If documents you bulk are big or with a high depth, you should send less docs per bulk or sending less bulk un parallel

7

Hi,

I also see my heap is growing and is not getting released at all as shown below. Why isn't GC getting kicked and releasing memory ?

jvm: {
timestamp: 1496287756707,
uptime: "15.8h",
uptime_in_millis: 56887012,
mem: {
heap_used: "5.8gb",
heap_used_in_bytes: 6229423928,
heap_used_percent: 99,
heap_committed: "5.8gb",
heap_committed_in_bytes: 6241845248,
heap_max: "5.8gb",
heap_max_in_bytes: 6241845248,
non_heap_used: "120.1mb",
non_heap_used_in_bytes: 125992016,
non_heap_committed: "123mb",
non_heap_committed_in_bytes: 128987136,
pools: {
young: {
used: "1.4gb",
used_in_bytes: 1605304320,
max: "1.4gb",
max_in_bytes: 1605304320,
peak_used: "1.4gb",
peak_used_in_bytes: 1605304320,
peak_max: "1.4gb",
peak_max_in_bytes: 1605304320
},
survivor: {
used: "179.4mb",
used_in_bytes: 188185344,
max: "191.3mb",
max_in_bytes: 200605696,
peak_used: "191.3mb",
peak_used_in_bytes: 200605696,
peak_max: "191.3mb",
peak_max_in_bytes: 200605696
},
old: {
used: "4.1gb",
used_in_bytes: 4435935184,
max: "4.1gb",
max_in_bytes: 4435935232,
peak_used: "4.1gb",
peak_used_in_bytes: 4435935232,
peak_max: "4.1gb",
peak_max_in_bytes: 4435935232
}
}
},
threads: {
count: 308,
peak_count: 322
},
gc: {
collectors: {
young: {
collection_count: 9192,
collection_time: "6.3m",
collection_time_in_millis: 379279
},
old: {
collection_count: 7715,
collection_time: "11.3h",
collection_time_in_millis: 40829356
}
}
}

8

I'm not an expert, but what I know is that theorically, Elasticsearch releases memory when it's needed.

Moreover, the way you request your indices has a real impact on heap use. If you make some aggregations or sorting very frequently, the heap will be full all the time.

Remember that you have only one node which must handle massive bulk operations, and probably some recurent requests which needs heap.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.