Elasticsearch indexing performance

Elastic Stack Elasticsearch
1

When indexing simple data of text type with elasticsearch, the performance difference is large according to each data size.

100byte -> 2000/s(0.2MB/s)
1KB -> 2000/s(2MB/s)
10KB -> 2000/s(20MB/s)

Why?
What is the reason?
Can't i speed up case of 100byte?

2

How are you sending data to Elasticsearch? How large bulk requests are you using? How many concurrent indexing threads/processes?

What is the specification of your cluster in terms of size and hardware?

3

How are you sending data to Elasticsearch?
app -> (file read)fluentd -> es

How large bulk requests are you using?
-> bulk 20MB

How many concurrent indexing threads/processes?
-> 3 processes(thread same)

What is the specification of your cluster in terms of size and hardware?
-> 4 GB MEM, 2 CPU

Regardless of the indexing method or hardware, this problem is the same.

4

How much data do you have built up on the source systems? How fast is data generated there? Does throughput vary service if you increase the number of clients workers?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.