Machine requirements for Elasticsearch

Elastic Stack Elasticsearch
1

I have an ES cluster with 3 nodes with 8GB RAM and 100GB HDD each and I've set their heap spaces to 4GB of RAM. They contain 8 indices with 70GB data combined and the complete data is updated daily at midnight. The problem that I'm facing is that while indexing the data, ES keeps going out of memory, throws an EngineClosedException and also the indexing script gives a Timeout error.

What I want to know is if the size of my machine is less for the given amount of data and indexing. If yes, how large a machine should I use in this situation?

EDIT

I'm sending the data to ES in batches of 5000 per request, I'm running a minimum of 20 threads at a time which I plan to increase to 40 by running two indexing scripts concurrently. I'm dumping the output of the indexing script in a file which gives me the information about timeout and when ES crashes, the log gives information about EngineClosedException and OutOfMemoryException.

Each batch of 5000 entries stands somewhere between 15-20 MB.

2

Hi @keshav

Could you provide more information around how you are indexing the data? Specifically, are you

  1. batching documents up into a certain mb payload sizes and sending as many bulk calls?
  2. sending bulk batches to Elasticsearch concurrently?
  3. observing the response of each bulk call?
3

Hi @forloop

I'm sending the data to ES in batches of 5000 per request, I'm running a minimum of 20 threads at a time which I plan to increase to 40 by running two indexing scripts concurrently. I'm dumping the output of the indexing script in a file which gives me the information about timeout and when ES crashes, the log gives information about EngineClosedException and OutOfMemoryException.

Each batch of 5000 entries stands somewhere between 15-20 MB.

4

Add more heap, add more nodes.

5

Given the relatively large batch size and the significant number of concurrent indexing threads I would say that you are overwhelming the cluster. Reduce the batch size (maybe to around 1000) and number of connections until it is stable and then gradually increase the number of connections until you see no further improvement in indexing throughput.

6

I did think of that and scaling vertically but I was wondering if I could make changes in my current configurations that could make the cluster more stable. If not I have been planning to scale up to nodes with 16GB RAM and 12GB heap space.

7

I can't reduce the batch size as on doing that the indexing becomes painstakingly slow. Can you suggest some change in the ES config though?

8

If you don't want to change the batch size, try reducing the number of indexing threads until it is stable. Then look at what is limiting performance. What does CPU usage and disk IO utilisation look like? How does the indexing rate at this stable level compare to what you see now?

There will be a limit to the performance of any cluster, so just increasing indexing threads is not likely to help. You may very well need to scale up or out.

Tuning will depend on which version of Elasticsearch you are using, which I did not see specified.