Es readtimeout occurs

Elastic Stack Elasticsearch
1

At 5:11, search readtimeout occurred a lot.

image

image

image
image1980×1052 180 KB

When we execute batch execution, if the primary shard is indexed, we are increasing the replica shard at that time, But at that time, the search engine creates replica shards, and as you can see from the metrix, a large amount of data is indexed, so the segment count rises rapidly.
So merge, flush, refresh spike...

Any good solutions?

2

By....? :wink:

3

@warkolm
sorry i removed that part
It has nothing to do with it, so ignore it and answer it! thanks

4

Can you clarify this, as it's mixing a few things and doesn't make sense to me sorry.

Batch execution?
"if the primary shard is indexed"?

5

@warkolm
once again
Our batch job increments the replica shard when the index on the primary shard is complete, but then the problem is that the replica shard is created too quickly.

Even if the time zone of the image above and the image below are different, the problem is the same, so you can ignore the time zone

image

image

image
image1156×271 36.8 KB

6

Why is that a problem?

Sorry, it's really not clear to me where this timeout you reference is coming from, nor what the problem you are seeing is. I understand you are showing us some graphs, but there's no context as to why you think these are causing issues, and where those issues are showing up.

7

@warkolm
Readtimeout occurs every time the i/o operation is high.
The reason why the i/o operation is high is when creating a replica shard...
When creating a replica shard, merge, refresh, and flush all increase, which causes readtimeout.

Do you understand now?

I need a solution for that

8

Where is the read timeout occuring?

9

@warkolm
Occurs in the api using the index shown in the metric

10

There's no timeout metric in those images.

11

@warkolm
Metric beat doesn't provide a timeout...

Are you saying you have no insight on this indicator?
So what do you need?? tell me that

image
image1153×806 108 KB

12

In the OP you state;

There are spikes in activity and other things, but if you are increasing replicas then that is to be expected.
However none of what you have provided shows a timeout.

So, where is this "search readtimeout" coming from if it's not in any of the metrics you have?

13

@warkolm

We cut the request from the api if it gets longer than 1 second
are you okay now?

[2022-03-10 07:17:15][/i-0a63522c348cba4fd.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:15][/i-0a96bdeb155a50a12.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:15][/i-0718f570d6728b257.][ERROR][xlog Error] : [hystrix]EsGoodsCountClient#EsGoodsCountClient - Read timed out
7:17
[2022-03-10 07:17:15][/i-0de020cdcf46d490c.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:15][/i-0a33d4d79a4c96630.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:15][/i-0a96bdeb155a50a12.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:15][/i-03237dc072449c3ee.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:16][/][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:16][/i-09d5625d41e21a8bb.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:16][/i-03237dc072449c3ee.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:16][/][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:16][/i-0718f570d6728b257.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:16][/i-0718f570d6728b257.][ERROR][xlog Error] : [hystrix]EsSearchTabClient#EsSearchTabClient - Read timed out
7:17
[2022-03-10 07:17:17][/i-0a96bdeb155a50a12.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:17][/i-0de020cdcf46d490c.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:17][/i-0d0ed434471719f8e.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:17][/i-0de020cdcf46d490c.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:17][/i-0718f570d6728b257.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:17][/i-03237dc072449c3ee.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:19][/i-0718f570d6728b257.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:19][/i-0de020cdcf46d490c.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:22][/i-0a63522c348cba4fd.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:26][/i-0d09f17015ba165ca.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:26][/][ERROR][xlog Error] : [hystrix]EsGoodsCountClient#EsGoodsCountClient - Read timed out
7:17
[2022-03-10 07:17:27][/i-0a33d4d79a4c96630.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:27][/][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:27][/i-0d09f17015ba165ca.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:27][/][ERROR][xlog Error] : [hystrix]EsGoodsCountClient#EsGoodsCountClient - Read timed out

@mayya help me