High elastic search heap memory consumption while indexing huge files

[akash (Akash Sudhakar)]

Hi,
We are trying to index large files in elastic search. Elastic heap memory consumption shoots up while indexing.

Basic observation is that while indexing memory can go upto 30x and post indexing it remains at around 10x.
For eg: while indexing 600MB file, memory usage is around 6GB and it can shoot upto 18 GB.
while indexing 1GB file, memory usage is around 10GB and it can shoot upto 30 GB.

Is this expected behaviour. Is there a way we can reduce the memory footprint.

Any help with this query is highly appreciated.

This behavior is seen in both ES 2.4 and ES 5.5 versions

2 data nodes, 1 dedicated master node.
ES Heap memory - 31 GB in all nodes

[Christian_Dahlqvist]

How are you indexing the data? How large are your documents? How large are your bulk requests? How many concurrent indexing threads do you use?

[akash (Akash Sudhakar)]

Hi Christian,

Please find details below -

How are you indexing the data?

We are calling bulk indexing API using curl command. We initially tried with transport client, but we were getting OutOfMemoryError.

How large are your documents?

We have tested with 600 MB and 1 GB data.

How large are your bulk requests?

We were trying to index a single document of 1 GB data. Document contains single key value pair.

How many concurrent indexing threads do you use?

Not sure about this. How to find this out

[Christian_Dahlqvist]

Questions around indexing very large documents have been asked before on the forum. Please have a look at:

Recommendation for indexing a large size document < 1G

Hi ES Team, This question may have been asked earlier. I am looking for ways to index really large documents like PDF, Word using ES. Our application crawls enterprise Active Directory system and we have hit some really large documents that we are unable to index as of now. We are seeing OutOfMemory Error in ES logs. I must also mention that for large files we pass a URL to a custom plugin running within ES server. This plugin reads the entire source file into memory from the provided URL befor…

Elastic search memory heap exception when trying to index large document by chunks

Hello there, Setup: the elastic search version that we are using is 2.3.5 we set es java heap size to be 4GB and then set bootstrap.mlockall: true UseCase: We have a use case where we would like to index very large document like 1GB using mapper-attachment plugin. We know that memory is going to be an issue so we are sending the 1 GB files into smaller chunks of 50MB into elastic search synchronously. To append the chunks to elastic i have used append script as shown below: var updateRe…

[akash (Akash Sudhakar)]

Above links are similar to our requirement where we need to index large documents. They are suggesting to split the document which would add complexity to search part.

Any idea why while indexing heap memory goes up to 30 x and post indexing it remains at around 10 x ??

Is there any settings we can change in order to reduce memory usage, say around 3 x or 4 x.

[Christian_Dahlqvist]

I am not surprised the heap usage grows a lot as there is a lot of analysis and processing going on. As mentioned in the links I provided, documents of that size is really beyond what Elasticsearch was designed for and you may also face issue searching them.

[akash (Akash Sudhakar)]

Ok Thanks a lot for the information.