Shards are going to Intialized state againa and again, like in every 15 mins

Elastic Stack Elasticsearch
1

My Elasticsearch cluster is going to yellow state in about every 15 min, becuase 2 replica shards are going to initialization state. After 5 mins or so, cluster is going back to green state.
Error Logs:

 [o.e.t.OutboundHandler    ] [SDV200000A] failed to serialize outbound message [Request{indices:data/write/bulk[s][r]}{19424480}{false}{false}{false}]
java.lang.IllegalArgumentException: ReleasableBytesStreamOutput cannot hold more than 2GB of data.

I checked the code also given in the stack trace

Caused by: java.lang.IllegalArgumentException: ReleasableBytesStreamOutput cannot hold more than 2GB of data
	at org.elasticsearch.common.io.stream.BytesStreamOutput.ensureCapacity(BytesStreamOutput.java:175)

ensureCapacity() is taking long argument and comparing its value with int_MAX32.
To my understanding it looks like it creates pages during serialization process and the total size of those create pages is going more than 2gbs.

Can you please share your thoughts around this and what could be the possible reasons for this issue?

And why its specifically happening for replica shards not for the primary one?

2

Which version of Elasticsearch are you using?

What is the size and specification of your cluster?

What type of hardware is the cluster deployed on?

3

ES7.17
This particular shard size is around 25GB, less that other shards for this index.
We are using Azure VMs as nodes.

4

It means a single bulk request is larger than 2GiB. That won't work, you need to send smaller bulk requests.

1 Like
5

No looks like that not the case here. We suspected this and also reduced our feeding batch size to 1GB but still happening.
Also if that would be the reason why its not happening on Primary shard, why only for replica shards.

6

1GiB is far too large for a single bulk request. The default limit for all HTTP requests is 100MiB, and the docs recommend you do not increase this. The primary may increase the sizes of the docs (e.g. via ingest pipelines).

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.