# Rolling upgrade: How many times should I flush? **URL:** \ **Category:** Elasticsearch\ **Created:** [January 12, 2021, 9:20pm UTC](https://discuss.elastic.co/t/rolling-upgrade-how-many-times-should-i-flush/260907 "2021-01-12T21:20:24Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Dennis1](https://avatars.discourse-cdn.com/v4/letter/d/ecccb3/32.png) [@Dennis1](https://discuss.elastic.co/u/Dennis1)\ **Post date:** [January 12, 2021, 9:20pm UTC](https://discuss.elastic.co/t/rolling-upgrade-how-many-times-should-i-flush/260907/1 "2021-01-12T21:20:24Z")
If I follow the steps in the [Rolling Upgrade docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/rolling-upgrades.html) in the order they're written, it seems to suggest performing a flush for every node I upgrade in my cluster. Is this really necessary, or would performing the flush once at the very beginning of the rolling upgrade probably be enough for purposes of speeding up shard recovery at the end of the whole process? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 12, 2021, 9:21pm UTC](https://discuss.elastic.co/t/rolling-upgrade-how-many-times-should-i-flush/260907/2 "2021-01-12T21:21:04Z")
If the indices are not being written to the flush won't be a major issue. How large is your cluster? ---
**Author:** ![Dennis1](https://avatars.discourse-cdn.com/v4/letter/d/ecccb3/32.png) [@Dennis1](https://discuss.elastic.co/u/Dennis1)\ **Post date:** [January 12, 2021, 9:23pm UTC](https://discuss.elastic.co/t/rolling-upgrade-how-many-times-should-i-flush/260907/3 "2021-01-12T21:23:56Z")
My cluster is 4 nodes (3 master-eligible, and 1 kibana/coordinating-only node). I can stop data ingestion to the cluster for the most part, but would likely (especially in the future) still have some metricbeat data trying to come in, which wouldn't be as feasible to turn off. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 12, 2021, 9:26pm UTC](https://discuss.elastic.co/t/rolling-upgrade-how-many-times-should-i-flush/260907/4 "2021-01-12T21:26:34Z")
> [@Dennis1](#): > > Is this really necessary No, as the docs say, it's optional. It makes shard recovery quicker, often saving more time than it takes to do the flush after each node, but it's definitely not necessary. ---
**Author:** ![Dennis1](https://avatars.discourse-cdn.com/v4/letter/d/ecccb3/32.png) [@Dennis1](https://discuss.elastic.co/u/Dennis1)\ **Post date:** [January 12, 2021, 9:29pm UTC](https://discuss.elastic.co/t/rolling-upgrade-how-many-times-should-i-flush/260907/5 "2021-01-12T21:29:58Z")
@DavidTurner Thank you. Yes sorry I shouldn't have used "necessary" in my original question--what I was more curious about is how much doing so will influence the speed of recovery at the end? In other words, if I flush just the first time, how much slower would recovery be than if I were to flush for every node? Would it even be noticeable? (Assuming a cluster size of \<10 nodes.) ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 12, 2021, 9:36pm UTC](https://discuss.elastic.co/t/rolling-upgrade-how-many-times-should-i-flush/260907/6 "2021-01-12T21:36:14Z")
If you're not indexing at all then it's a wash: it won't make recovery any faster but nor will it take any time to do the flush for each node. If you carry on indexing then I'd usually expect repeated flushes to save time overall — that's why it's recommended — but the time saved is going to depend on how hard you're indexing and how your shards are allocated. There's no definite answer, sorry, you'll have to experiment. Maybe we should turn the question around: why is it proving difficult for you to flush as recommended? ---
**Author:** ![Dennis1](https://avatars.discourse-cdn.com/v4/letter/d/ecccb3/32.png) [@Dennis1](https://discuss.elastic.co/u/Dennis1)\ **Post date:** [January 12, 2021, 9:43pm UTC](https://discuss.elastic.co/t/rolling-upgrade-how-many-times-should-i-flush/260907/7 "2021-01-12T21:43:14Z")
@DavidTurner (and @warkolm) Thank you! That's all I was basically interested in—i.e., what factors to be aware of that may make it more valuable to re-flush each time or not. It sounds like the main factor would be whether I'm still indexing or not, and the volume of indexing that's occurring over the course of the upgrade (which itself relates to the number of nodes in my cluster that need to be upgraded). I'm still just getting ready for a rolling upgrade, but I'll probably opt to flush each time just to prepare for scaling in the future. ---
**Author:** ![kkn87](https://avatars.discourse-cdn.com/v4/letter/k/cab0a1/32.png) [@kkn87](https://discuss.elastic.co/u/kkn87)\ **Post date:** [January 14, 2021, 6:37am UTC](https://discuss.elastic.co/t/rolling-upgrade-how-many-times-should-i-flush/260907/8 "2021-01-14T06:37:01Z")
We do rolling upgrades automatically, using Ansible. Just added /flush task to the script before each node's update. Also, at start, we've added changing 'index.unassigned.node\_left.delayed\_timeout' to 10 minutes on all indices to make recovers a bit faster after node restart. Using this approach, rolling upgrade of our cluster (6 data nodes, 20 TB of data) takes 2 hours. It includes OS updates and nodes restarts. ---
**Author:** ![Dennis1](https://avatars.discourse-cdn.com/v4/letter/d/ecccb3/32.png) [@Dennis1](https://discuss.elastic.co/u/Dennis1)\ **Post date:** [January 14, 2021, 4:27pm UTC](https://discuss.elastic.co/t/rolling-upgrade-how-many-times-should-i-flush/260907/9 "2021-01-14T16:27:33Z")
@kkn87 Thanks for the tips and sharing your approach/experience! I'm curious on why increasing the `index.unassigned.node_left.delayed_timeout` to 10 mins proved to speed up the recovery process. Are you choosing to increase that timeout parameter as opposed to disabling replica shard allocation? ---
**Author:** ![kkn87](https://avatars.discourse-cdn.com/v4/letter/k/cab0a1/32.png) [@kkn87](https://discuss.elastic.co/u/kkn87)\ **Post date:** [January 15, 2021, 10:00am UTC](https://discuss.elastic.co/t/rolling-upgrade-how-many-times-should-i-flush/260907/10 "2021-01-15T10:00:24Z")
@Dennis1, you suggest not to do this, because when shard allocation is disabled (swtiched to 'primaries') - changing index.unassigned.node\_left.delayed\_timeout is obsolete? Maybe I misunderstood this article [https://www.elastic.co/guide/en/elasticsearch/reference/current/delayed-allocation.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/delayed-allocation.html) then. ---
**Author:** ![Dennis1](https://avatars.discourse-cdn.com/v4/letter/d/ecccb3/32.png) [@Dennis1](https://discuss.elastic.co/u/Dennis1)\ **Post date:** [January 15, 2021, 2:16pm UTC](https://discuss.elastic.co/t/rolling-upgrade-how-many-times-should-i-flush/260907/11 "2021-01-15T14:16:18Z")
@kkn87 Yes that's my understanding at least. The only difference between increasing `index.unassigned.node_left.delayed_timeout` versus disabling and then re-enabling `cluster.routing.allocation.enable` (as in steps 1 & 9 [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/rolling-upgrades.html#_upgrading_your_cluster)) is that the latter gives time for the re-balancing (or "recovery") to occur when each node is brought back online after it's been upgraded. If the delay time is kept at 10 mins for the entire upgrade process, that recovery step may never occur, which I guess is not recommended especially in cases where there is a lot of indexing still happening during the upgrade. ---
**Author:** ![kkn87](https://avatars.discourse-cdn.com/v4/letter/k/cab0a1/32.png) [@kkn87](https://discuss.elastic.co/u/kkn87)\ **Post date:** [January 18, 2021, 7:38am UTC](https://discuss.elastic.co/t/rolling-upgrade-how-many-times-should-i-flush/260907/12 "2021-01-18T07:38:23Z")
@Dennis1, thank you for noticing this! Will remove this step from the automation. ---
**Author:** ![Dennis1](https://avatars.discourse-cdn.com/v4/letter/d/ecccb3/32.png) [@Dennis1](https://discuss.elastic.co/u/Dennis1)\ **Post date:** [January 19, 2021, 4:49pm UTC](https://discuss.elastic.co/t/rolling-upgrade-how-many-times-should-i-flush/260907/13 "2021-01-19T16:49:16Z")
@kkn87 Sure thing, just make sure to add some sort of automated check for the cluster status to return back to 'green' after upgrading each node, before moving onto the next node. The green status indicates that the recovery process is finished. ---
**Author:** ![kkn87](https://avatars.discourse-cdn.com/v4/letter/k/cab0a1/32.png) [@kkn87](https://discuss.elastic.co/u/kkn87)\ **Post date:** [January 20, 2021, 7:42am UTC](https://discuss.elastic.co/t/rolling-upgrade-how-many-times-should-i-flush/260907/14 "2021-01-20T07:42:42Z")
Sure, @Dennis1, we have quite comprehensive checks, waiting for completely 'green' cluster: url: '{{ elasticsearch\_url }}/\_cluster/health' ... until: \> result.status == 200 and result.json.status == 'green' and result.json.unassigned\_shards == 0 and result.json.relocating\_shards == 0 and result.json.number\_of\_nodes == previous\_result.number\_of\_nodes ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 17, 2021, 7:42am UTC](https://discuss.elastic.co/t/rolling-upgrade-how-many-times-should-i-flush/260907/15 "2021-02-17T07:42:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.