# Shard check on startup is slow **URL:** \ **Category:** Elasticsearch\ **Created:** [November 3, 2011, 9:01am UTC](https://discuss.elastic.co/t/shard-check-on-startup-is-slow/5759 "2011-11-03T09:01:02Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![vpunski](https://avatars.discourse-cdn.com/v4/letter/v/54ee81/32.png) [@vpunski](https://discuss.elastic.co/u/vpunski)\ **Post date:** [November 3, 2011, 9:01am UTC](https://discuss.elastic.co/t/shard-check-on-startup-is-slow/5759/1 "2011-11-03T09:01:02Z")
I tried to use index.shard.check\_on\_startup in order to check the consistency of shard. For unknown reason, this process is very slow. From short investigation of system resources, I see that CPU usage is very low, but it doesn't seems to be IO intensive. My discs read about 20MB/sec, when using "hdparam -t /dev/sda" it gets about 100MB/sec. What is the reason of so slow loading? Current shard configuration: { "cluster\_name" : "MY\_CLUSTER", "status" : "green", "timed\_out" : false, "number\_of\_nodes" : 10, "number\_of\_data\_nodes" : 10, "active\_primary\_shards" : 10, "active\_shards" : 30, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 0 } ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 3, 2011, 5:15pm UTC](https://discuss.elastic.co/t/shard-check-on-startup-is-slow/5759/2 "2011-11-03T17:15:29Z")
I mentioned before that this is a slow process. It needs to check and read quite a lot of data from the index. On Thu, Nov 3, 2011 at 11:01 AM, vadim [vpunski@gmail.com](mailto:vpunski@gmail.com) wrote: > I tried to use index.shard.check\_on\_startup in order to check the > consistency of shard. > For unknown reason, this process is very slow. > From short investigation of system resources, I see that CPU usage is > very low, but it doesn't seems to be IO intensive. > My discs read about 20MB/sec, when using "hdparam -t /dev/sda" it gets > about 100MB/sec. > > What is the reason of so slow loading? > > Current shard configuration: > { > "cluster\_name" : "MY\_CLUSTER", > "status" : "green", > "timed\_out" : false, > "number\_of\_nodes" : 10, > "number\_of\_data\_nodes" : 10, > "active\_primary\_shards" : 10, > "active\_shards" : 30, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 0 > > } ---
**Author:** ![vpunski](https://avatars.discourse-cdn.com/v4/letter/v/54ee81/32.png) [@vpunski](https://discuss.elastic.co/u/vpunski)\ **Post date:** [November 4, 2011, 8:20am UTC](https://discuss.elastic.co/t/shard-check-on-startup-is-slow/5759/3 "2011-11-04T08:20:10Z")
Yes, you've mentioned it before. Now I'd like to understand the actual reason of slow read rate (from IO perspective), not the "slow process". By reason I mean for example non sequential file read requiered by Lucene index check process. Anyway, I'd like to understand the real reason and try to solve it. From what I've seen in the code, as all the shards started in parallel, their "check on startup" process creates race condition on hardware storage (disc). Am I missing something. Thanks On Nov 3, 2011 7:15 PM, "Shay Banon" [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > I mentioned before that this is a slow process. It needs to check and read > quite a lot of data from the index. > > On Thu, Nov 3, 2011 at 11:01 AM, vadim [vpunski@gmail.com](mailto:vpunski@gmail.com) wrote: > > > I tried to use index.shard.check\_on\_startup in order to check the > > consistency of shard. > > For unknown reason, this process is very slow. > > From short investigation of system resources, I see that CPU usage is > > very low, but it doesn't seems to be IO intensive. > > My discs read about 20MB/sec, when using "hdparam -t /dev/sda" it gets > > about 100MB/sec. > > > > What is the reason of so slow loading? > > > > Current shard configuration: > > { > > "cluster\_name" : "MY\_CLUSTER", > > "status" : "green", > > "timed\_out" : false, > > "number\_of\_nodes" : 10, > > "number\_of\_data\_nodes" : 10, > > "active\_primary\_shards" : 10, > > "active\_shards" : 30, > > "relocating\_shards" : 0, > > "initializing\_shards" : 0, > > "unassigned\_shards" : 0 > > > > } ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 8, 2011, 3:01am UTC](https://discuss.elastic.co/t/shard-check-on-startup-is-slow/5759/4 "2011-11-08T03:01:13Z")
You can have a look at the CheckIndex class in Lucene to see how its implemented. Regarding the number of shards started in parallel on a node, you can control that. By default, it will allow for 4 primary shards to be started (cluster.routing.allocation.node\_initial\_primaries\_recoveries setting) in parallel on a node, and 2 replica shards (cluster.routing.allocation.node\_concurrent\_recoveries setting). On Fri, Nov 4, 2011 at 10:20 AM, Vadim Punski [vpunski@gmail.com](mailto:vpunski@gmail.com) wrote: > Yes, you've mentioned it before. Now I'd like to understand the actual > reason of slow read rate (from IO perspective), not the "slow process". By > reason I mean for example non sequential file read requiered by Lucene > index check process. Anyway, I'd like to understand the real reason and try > to solve it. > > From what I've seen in the code, as all the shards started in parallel, > their "check on startup" process creates race condition on hardware storage > (disc). > > Am I missing something. > > Thanks > On Nov 3, 2011 7:15 PM, "Shay Banon" [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > I mentioned before that this is a slow process. It needs to check and > > read quite a lot of data from the index. > > > > On Thu, Nov 3, 2011 at 11:01 AM, vadim [vpunski@gmail.com](mailto:vpunski@gmail.com) wrote: > > > > > I tried to use index.shard.check\_on\_startup in order to check the > > > consistency of shard. > > > For unknown reason, this process is very slow. > > > From short investigation of system resources, I see that CPU usage is > > > very low, but it doesn't seems to be IO intensive. > > > My discs read about 20MB/sec, when using "hdparam -t /dev/sda" it gets > > > about 100MB/sec. > > > > > > What is the reason of so slow loading? > > > > > > Current shard configuration: > > > { > > > "cluster\_name" : "MY\_CLUSTER", > > > "status" : "green", > > > "timed\_out" : false, > > > "number\_of\_nodes" : 10, > > > "number\_of\_data\_nodes" : 10, > > > "active\_primary\_shards" : 10, > > > "active\_shards" : 30, > > > "relocating\_shards" : 0, > > > "initializing\_shards" : 0, > > > "unassigned\_shards" : 0 > > > > > > } ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:49am UTC](https://discuss.elastic.co/t/shard-check-on-startup-is-slow/5759/5 "2017-07-06T03:49:41Z")