At 5:11, search readtimeout occurred a lot.
When we execute batch execution, if the primary shard is indexed, we are increasing the replica shard at that time, But at that time, the search engine creates replica shards, and as you can see from the metrix, a large amount of data is indexed, so the segment count rises rapidly.
So merge, flush, refresh spike...
Any good solutions?
By....? 
Can you clarify this, as it's mixing a few things and doesn't make sense to me sorry.
Batch execution?
"if the primary shard is indexed"?
@warkolm
once again
Our batch job increments the replica shard when the index on the primary shard is complete, but then the problem is that the replica shard is created too quickly.
Even if the time zone of the image above and the image below are different, the problem is the same, so you can ignore the time zone
Why is that a problem?
Sorry, it's really not clear to me where this timeout you reference is coming from, nor what the problem you are seeing is. I understand you are showing us some graphs, but there's no context as to why you think these are causing issues, and where those issues are showing up.
@warkolm
Readtimeout occurs every time the i/o operation is high.
The reason why the i/o operation is high is when creating a replica shard...
When creating a replica shard, merge, refresh, and flush all increase, which causes readtimeout.
Do you understand now?
I need a solution for that
Where is the read timeout occuring?
There's no timeout metric in those images.
@warkolm
Metric beat doesn't provide a timeout...
Are you saying you have no insight on this indicator?
So what do you need?? tell me that
In the OP you state;
There are spikes in activity and other things, but if you are increasing replicas then that is to be expected.
However none of what you have provided shows a timeout.
So, where is this "search readtimeout" coming from if it's not in any of the metrics you have?
We cut the request from the api if it gets longer than 1 second
are you okay now?
[2022-03-10 07:17:15][/i-0a63522c348cba4fd.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:15][/i-0a96bdeb155a50a12.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:15][/i-0718f570d6728b257.][ERROR][xlog Error] : [hystrix]EsGoodsCountClient#EsGoodsCountClient - Read timed out
7:17
[2022-03-10 07:17:15][/i-0de020cdcf46d490c.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:15][/i-0a33d4d79a4c96630.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:15][/i-0a96bdeb155a50a12.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:15][/i-03237dc072449c3ee.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:16][/][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:16][/i-09d5625d41e21a8bb.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:16][/i-03237dc072449c3ee.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:16][/][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:16][/i-0718f570d6728b257.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:16][/i-0718f570d6728b257.][ERROR][xlog Error] : [hystrix]EsSearchTabClient#EsSearchTabClient - Read timed out
7:17
[2022-03-10 07:17:17][/i-0a96bdeb155a50a12.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:17][/i-0de020cdcf46d490c.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:17][/i-0d0ed434471719f8e.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:17][/i-0de020cdcf46d490c.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:17][/i-0718f570d6728b257.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:17][/i-03237dc072449c3ee.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:19][/i-0718f570d6728b257.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:19][/i-0de020cdcf46d490c.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:22][/i-0a63522c348cba4fd.][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:26][/i-0d09f17015ba165ca.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:26][/][ERROR][xlog Error] : [hystrix]EsGoodsCountClient#EsGoodsCountClient - Read timed out
7:17
[2022-03-10 07:17:27][/i-0a33d4d79a4c96630.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:27][/][ERROR][xlog Error] : [hystrix]GoodsSearchClient#GoodsSearchClient - Read timed out
7:17
[2022-03-10 07:17:27][/i-0d09f17015ba165ca.][ERROR][xlog Error] : [hystrix]GoodsClient#GoodsClient - Read timed out
7:17
[2022-03-10 07:17:27][/][ERROR][xlog Error] : [hystrix]EsGoodsCountClient#EsGoodsCountClient - Read timed out
@mayya help me
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.
