Write slow on elastic cluster

Elastic Stack Elasticsearch
1

Hi All

One of clusters has the following specs
8 nodes (8 data and 5 master eligible) each with 15GB heap space running on servers with 32GB of RAM.
Elasticsearch version 7.6.2 on linux
each node has 8 TB of space allocated
around 30 billion documents are there
Each node has shards of 1443
total write i/o is around 350/sec
The application team is complaining the writes are slow, how can we improve writes to cluster?

Thanks
Sunil

2

That looks like a lot of shards per node. Significantly more than recommended. How many indices and shards are you actively indexing into? How are you indexing into Elasticsearch? What type of storage do you have?

3

Hi Christian

storage is XIO Flash storage
ingestion happening from a python script, any given time there are 34 processes that are capable of writing.
each process will only write to a single index at a time, index refresh is disabled when they start on a particular work item
on completion of the work item they trigger a manual refresh

the volume of 2.2k items per day will be roughly ingested, which isn't 1:1 with indexes (As several of these items share an index)

Thanks
Sunil

4

What bulk size are you using?

5

Hi Christian

500 events/chunk, max size of 50MB , 4 queues

Thanks
Sunil

6

What is the average size of your events? Are you using dynamic mappings?

7

That's too many, you need to at least halve that.

8

Hi Christian

average size of event is around 3kb and dynamic sampling is being used

Thanks
Sunil

9

What unit is this? What ingest rate are you seeing?

10

Hi warkolm

do we need to add additional nodes? or increase in memory? our infrastructure team has said even half of the given memory is not getting allocated.

Thanks
Sunil

11

Hi Christian

i/o statistics captured from kibana console and ingestion rate is 1.2 k events per second

Thanks
Sunil

12

How have you determined that it is the Elasticsearch cluster that is the bottleneck and not the Python scripts you are using to ingest the data?

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.