Slow Elasticsearch Ingestion

Elastic Stack Elasticsearch
1

I am new to ELK stack and I am currently testing on the performance of the Elasticsearch with 3 different configurations as follows:

  1. One source file: 1GB
  2. One source file: 3GB
  3. 3 source files: 1GB+1GB+1GB

It took me 3 to 4 hours to have the 3GB file ingested into Elasticsearch. It took me more than a day to finish 3 different 1GB files to be ingested. The speed was getting slower as the size of the index increases.

I have tried to increase the JVM heap size and it's getting slightly better, shortening the time by about one hour. All other settings are configured in the default settings.

I would like to know the additional ways on how to improve the ingestion speed on this? Thanks.

2

What is the specification of your cluster in terms of nodes and hardware used How are you infesting the data?

3

This is my current configuration of Elasticsearch. All are in default settings.
image

4

What is the specification of the host? What type of disk/storage do you have?

5

I am using VM here in an ESXI server so the hardware specification is as follows:
image

As I need this ELK for only one application so I didn't allocate high hard spec for this VM.

6

Indexing is I/O intensive so having fast storage is important for optimal indexing speed. I would recommend running ‘iostat -x’ on the node while indexing.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.