slowup
(nic k)
March 8, 2022, 12:12pm
1
At 5:11, search readtimeout occurred a lot.
When we execute batch execution, if the primary shard is indexed, we are increasing the replica shard at that time, But at that time, the search engine creates replica shards, and as you can see from the metrix, a large amount of data is indexed, so the segment count rises rapidly.
So merge, flush, refresh spike...
Any good solutions?
slowup
(nic k)
March 9, 2022, 4:14am
3
@warkolm
sorry i removed that part
It has nothing to do with it, so ignore it and answer it! thanks
warkolm
(Mark Walkom)
March 9, 2022, 4:24am
4
Can you clarify this, as it's mixing a few things and doesn't make sense to me sorry.
Batch execution?
"if the primary shard is indexed"?
slowup
(nic k)
March 10, 2022, 12:50am
5
@warkolm
once again
Our batch job increments the replica shard when the index on the primary shard is complete, but then the problem is that the replica shard is created too quickly.
Even if the time zone of the image above and the image below are different, the problem is the same, so you can ignore the time zone
warkolm
(Mark Walkom)
March 10, 2022, 2:39am
6
Why is that a problem?
Sorry, it's really not clear to me where this timeout you reference is coming from, nor what the problem you are seeing is. I understand you are showing us some graphs, but there's no context as to why you think these are causing issues, and where those issues are showing up.
slowup
(nic k)
March 10, 2022, 2:49am
7
@warkolm
Readtimeout occurs every time the i/o operation is high.
The reason why the i/o operation is high is when creating a replica shard...
When creating a replica shard, merge, refresh, and flush all increase, which causes readtimeout.
Do you understand now?
I need a solution for that
warkolm
(Mark Walkom)
March 10, 2022, 2:51am
8
Where is the read timeout occuring?
slowup
(nic k)
March 10, 2022, 2:53am
9
@warkolm
Occurs in the api using the index shown in the metric
warkolm
(Mark Walkom)
March 10, 2022, 2:56am
10
There's no timeout metric in those images.
slowup
(nic k)
March 10, 2022, 3:03am
11
@warkolm
Metric beat doesn't provide a timeout...
Are you saying you have no insight on this indicator?
So what do you need?? tell me that
warkolm
(Mark Walkom)
March 10, 2022, 3:07am
12
In the OP you state;
There are spikes in activity and other things, but if you are increasing replicas then that is to be expected.
However none of what you have provided shows a timeout.
So, where is this "search readtimeout" coming from if it's not in any of the metrics you have?
slowup
(nic k)
March 10, 2022, 3:18am
13
@warkolm
We cut the request from the api if it gets longer than 1 second
are you okay now?
[2022-03-10 07:17:15][/i-0a63522c348cba4fd.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:15][/i-0a96bdeb155a50a12.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:15][/i-0718f570d6728b257.][ERROR][xlog Error] : [hystrix]EsGoodsCountClient#EsGoodsCountClient - Read timed out
7:17
[2022-03-10 07:17:15][/i-0de020cdcf46d490c.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:15][/i-0a33d4d79a4c96630.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:15][/i-0a96bdeb155a50a12.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:15][/i-03237dc072449c3ee.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:16][/][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:16][/i-09d5625d41e21a8bb.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:16][/i-03237dc072449c3ee.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:16][/][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:16][/i-0718f570d6728b257.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:16][/i-0718f570d6728b257.][ERROR][xlog Error] : [hystrix]EsSearchTabClient#EsSearchTabClient - Read timed out
7:17
[2022-03-10 07:17:17][/i-0a96bdeb155a50a12.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:17][/i-0de020cdcf46d490c.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:17][/i-0d0ed434471719f8e.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:17][/i-0de020cdcf46d490c.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:17][/i-0718f570d6728b257.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:17][/i-03237dc072449c3ee.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:19][/i-0718f570d6728b257.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:19][/i-0de020cdcf46d490c.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:22][/i-0a63522c348cba4fd.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:26][/i-0d09f17015ba165ca.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:26][/][ERROR][xlog Error] : [hystrix]EsGoodsCountClient#EsGoodsCountClient - Read timed out
7:17
[2022-03-10 07:17:27][/i-0a33d4d79a4c96630.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:27][/][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:27][/i-0d09f17015ba165ca.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:27][/][ERROR][xlog Error] : [hystrix]EsGoodsCountClient#EsGoodsCountClient - Read timed out
@mayya help me
system
(system)
Closed
April 7, 2022, 3:18am
14
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.