# Very low speed resync after node stop/start
**URL:** https://discuss.elastic.co/t/very-low-speed-resync-after-node-stop-start/72122
**Category:** Elasticsearch
**Created:** [January 19, 2017, 6:56am UTC](https://discuss.elastic.co/t/very-low-speed-resync-after-node-stop-start/72122 "2017-01-19T06:56:06Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@Electron](https://discuss.elastic.co/u/Electron)
#### Post date: [January 19, 2017, 6:56am UTC](https://discuss.elastic.co/t/very-low-speed-resync-after-node-stop-start/72122/1 "2017-01-19T06:56:06Z")
Hello,
I have 5 elasticsearch nodes in one cluster ( near 85.000.000 docs , 30Gb data ). After stop and start one server I saw, that resync started with very low speed and very high load at each of cluster nodes. Randomly that nodes go out of cluster and resync start again. What I can do for fix that trouble ?
{
"cluster\_name" : "Prod",
"status" : "red",
"timed\_out" : false,
"number\_of\_nodes" : 4,
"number\_of\_data\_nodes" : 4,
"active\_primary\_shards" : 1863,
"active\_shards" : 3294,
"relocating\_shards" : 0,
"initializing\_shards" : 20,
"unassigned\_shards" : 6201,
"delayed\_unassigned\_shards" : 0,
"number\_of\_pending\_tasks" : 818,
"number\_of\_in\_flight\_fetch" : 0,
"task\_max\_waiting\_in\_queue\_millis" : 4071620,
"active\_shards\_percent\_as\_number" : 34.619022595901214
}
elasticsearch:
build: ./elastic
container\_name: elastic
command: elasticsearch -Des.network.host=0.0.0.0
net: host
ports:
- "9200:9200"
- "9300:9300"
volumes:
- "/srv/docker/elastic/etc:/usr/share/elasticsearch/config"
- "/srv/docker/elastic/db:/usr/share/elasticsearch/data"
- "/srv/backup/elasticsearch:/backup"
restart: always
[cluster.name](http://cluster.name): Prod
[node.name](http://node.name): "node04"
http.port: 9200
network.host: _non\_loopback_
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: [
"node02",
"node03",
"node04",
"node05",
"node06"
]
transport.publish\_host: 0.0.0.0
http.cors.enabled : true
http.cors.allow-origin : "\*"
http.cors.allow-methods : OPTIONS, HEAD, GET, POST, PUT, DELETE
http.cors.allow-headers : X-Requested-With,X-Auth-Token,Content-Type, Content-Length
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [January 19, 2017, 7:54am UTC](https://discuss.elastic.co/t/very-low-speed-resync-after-node-stop-start/72122/2 "2017-01-19T07:54:22Z")
You have 9 495 shards on 5 nodes?
Which is about 2000 shards per node.
Would you run 2000 databases instances on one single physical machine?
That's a lot.
You did not give your version BTW. If you did not upgrade, upgrade.
Reduce the number of shards. May be you have 5 shards per index and 1 replica? If not needed, reduce that number.
You have here `"number_of_pending_tasks" : 818`. So I think you have to wait it recovers.
Also look at your logs. They will probably tell you what happened.
---
### Author:  [@Electron](https://discuss.elastic.co/u/Electron)
#### Post date: [January 19, 2017, 8:05am UTC](https://discuss.elastic.co/t/very-low-speed-resync-after-node-stop-start/72122/3 "2017-01-19T08:05:49Z")
Hello, thank you for response.
My dockerfile is
FROM elasticsearch:latest
RUN if [! -d /usr/share/elasticsearch/plugins/hq]; then /usr/share/elasticsearch/bin/plugin install royrusso/elasticsearch-HQ; fi
RUN if [! -d /usr/share/elasticsearch/plugins/kopf]; then /usr/share/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf/2.0; fi
my replica set is 4. We need all of indexes at all of servers.
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [January 19, 2017, 9:51am UTC](https://discuss.elastic.co/t/very-low-speed-resync-after-node-stop-start/72122/4 "2017-01-19T09:51:17Z")
But why so many primary shards then?
9 495 / 5 = 1899 primary shards. May be one shard per index: 1899 indices...
85.000.000 / 1899 = 44k docs per primary shard.
30 Gb / 1899 = 15 mb per primary shard
You can probably go up to 20 gb per shard. That's a lot of waste IMO.
---
### Author:  [@Electron](https://discuss.elastic.co/u/Electron)
#### Post date: [January 19, 2017, 10:40am UTC](https://discuss.elastic.co/t/very-low-speed-resync-after-node-stop-start/72122/5 "2017-01-19T10:40:39Z")
I solve my problem by double heap size at each node ( from 1g to 2g ) , cluster wasn't stuck at resync at done it well. Thank you.
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [January 19, 2017, 11:02am UTC](https://discuss.elastic.co/t/very-low-speed-resync-after-node-stop-start/72122/6 "2017-01-19T11:02:42Z")
Great but still. Reduce the number of shards.
---
### Author:  [@Electron](https://discuss.elastic.co/u/Electron)
#### Post date: [January 19, 2017, 11:07am UTC](https://discuss.elastic.co/t/very-low-speed-resync-after-node-stop-start/72122/7 "2017-01-19T11:07:04Z")
This time have :
{
"cluster\_name" : "Prod",
"status" : "green",
"timed\_out" : false,
"number\_of\_nodes" : 5,
"number\_of\_data\_nodes" : 5,
"active\_primary\_shards" : 1918,
"active\_shards" : 9515,
"relocating\_shards" : 0,
"initializing\_shards" : 0,
"unassigned\_shards" : 0,
"delayed\_unassigned\_shards" : 0,
"number\_of\_pending\_tasks" : 0,
"number\_of\_in\_flight\_fetch" : 0,
"task\_max\_waiting\_in\_queue\_millis" : 0,
"active\_shards\_percent\_as\_number" : 100.0
}
with 30Gb data, 85.000.000 docs and 630 indicies. It is not optimal ? Indexes mask like log-`date`
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [January 19, 2017, 2:13pm UTC](https://discuss.elastic.co/t/very-low-speed-resync-after-node-stop-start/72122/8 "2017-01-19T14:13:42Z")
Those are logs?
Why do you need 4 replicas then?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [January 19, 2017, 2:15pm UTC](https://discuss.elastic.co/t/very-low-speed-resync-after-node-stop-start/72122/9 "2017-01-19T14:15:04Z")
Considering that you only have 30GB of data, you have far too many indices and shards. With this amount of data each daily index should only have a single primary shard, and I would most likely recommend switching to e.g. monthly indices in order to increase the average shard size and thereby reduce the number of induces/shards that need to be managed and the overhead associated with these. If you are on Elasticsearch 5.x, you can use the [shrink index API](https://www.elastic.co/guide/en/elasticsearch/reference/5.1/indices-shrink-index.html) to get down to 1 primary shard per index. Even though it is getting a bit old, [this blog post](https://www.elastic.co/blog/found-crash-elasticsearch) also contains some good points.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 16, 2017, 2:15pm UTC](https://discuss.elastic.co/t/very-low-speed-resync-after-node-stop-start/72122/10 "2017-02-16T14:15:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.