# Replicas synchronization

**URL:** <https://discuss.elastic.co/t/replicas-synchronization/37091>\
**Category:** Elasticsearch\
**Created:** [December 14, 2015, 6:43am UTC](https://discuss.elastic.co/t/replicas-synchronization/37091 "2015-12-14T06:43:10Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Crassirostris](https://avatars.discourse-cdn.com/v4/letter/c/779978/32.png) [@Crassirostris](https://discuss.elastic.co/u/Crassirostris)\
**Post date:** [December 14, 2015, 6:43am UTC](https://discuss.elastic.co/t/replicas-synchronization/37091/1 "2015-12-14T06:43:10Z")

</div>

Right now we're able to recreate situation with ES 1.6 when short connectivity problem (not long enough for node to be disconnected from cluster) leads to temporary replicas desynchronization. During short period of time it is possible for used to get different results with the same search query. In our project we need to be absolutely sure that no such divergence appears.

To prevent this kind of situations we need a way to understand that all replicas are in sync.

We considered using document version, but we can't find a way to get document version from a specific (or from every) shard replica. Another idea was using shard versions, but we use NEST client, it's not released for ES 2.0 yet. As far as I know, shard version is only available via REST API since ES 2.0. And I'm not even sure this approach works since version is not incremented per operation.

So the question remains — is there a way to confirm replicas synchronization, even with a performance drawback?

Additionally, is there a way to get document from a specific replica?

---

<div class="post-metadata">

**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\
**Post date:** [December 14, 2015, 7:18pm UTC](https://discuss.elastic.co/t/replicas-synchronization/37091/2 "2015-12-14T19:18:03Z")

</div>

Two potential solutions.

If your indexing is relatively light, it'd probably be easiest to check the `sync_id` marker on the indices. [Synced flush API | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-synced-flush.html)

Synced Flush was added in 1.6 (although I don't remember which patch version, you'd have to check to see if your 1.6 supports it). When indexing halts for 5 minutes, the shards will synchronize and set a new "commit point" which is represented by the `sync_id` marker. If you compare the primary vs replica's `sync_marker`, you can determine if they are synced up.

However, this only works if your indexing isn't continuous, and ES has a chance to execute a synced flush.

Alternatively, you can ask ES for the "authoritative copy" of the document by always going to the primary shard. This can be done using the `preference` parameter: [Request body search | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-preference.html)

You can us the `preference` on all read requests (search, GET, etc). To always go to the primary, you'd do something like:

```auto
GET /_search?preference=_primary
{
    "query": {
        "match": {
            "title": "elasticsearch"
        }
    }
}

```

The downside is fairly obvious: you'll always be hitting the primary shard to get the data. So it is best to use this only when necessary, and allow your non-critical queries to hit the replicas. But this is really the only way to guarantee that a client will see the most up-to-date copy, since the primary will always represent the authoritative, most recent version of the data.

> [@Crassirostris](#):
>
> Additionally, is there a way to get document from a specific replica?

Yep, using the same `preference` param, you can specify `?preference=_shards:2` to get from the second shard, or something like `?preference=_only_node:xyz` to get the shard from the node `xyz`, etc etc

---

<div class="post-metadata">

**Author:** ![Crassirostris](https://avatars.discourse-cdn.com/v4/letter/c/779978/32.png) [@Crassirostris](https://discuss.elastic.co/u/Crassirostris)\
**Post date:** [December 15, 2015, 3:56am UTC](https://discuss.elastic.co/t/replicas-synchronization/37091/3 "2015-12-15T03:56:37Z")

</div>

Thank you very much for your answer!

We use sync flush heavily for our cluster restart, thank you for that feature, without it, restarting our 20 TB cluster would be total pain. But I totally missed the sync\_marker.

Sync flush may do the deal, we only have direct writes to ES rarely, most of the data gets there via continuously working indexer that sends data in large bulks. I will try to fire sync flush request between two bulks and see if it works and if works fast enough.

Totally missed that preference too! So ashamed of myself right now, thank you a lot for clearing that out 🙂 Next time I totally re-check all the documentation before asking a question.

Great to see Elasticsearch grow and spread, you guys really do an amazing job!

By the way, right now ES doesn't store cluster state and that may lead to the situation, when after restart data was restored from stale replicas, because ones with the newer version didn't start yet. Are there any plans to fix it somehow or is it just the way ES is? Because obviously, persistent cluster state absence has it's advantages like flexibility. Since ES is all about being elastic, keeping it that way would be understandable.

[https://www.elastic.co/guide/en/elasticsearch/guide/current/\_important\_configuration\_changes.html#\_recovery\_settings](https://www.elastic.co/guide/en/elasticsearch/guide/current/_important_configuration_changes.html#_recovery_settings)  
Maybe I missed something in the docs again? Right now we use preferences from the page above, all kinds of `recovery_after...` but that wouldn't help us is case when ES has started and index request was fired. That would increment running shards versions and automatically made those not started outdated, even if they have data current shards do not.

---

<div class="post-metadata">

**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\
**Post date:** [December 15, 2015, 1:16pm UTC](https://discuss.elastic.co/t/replicas-synchronization/37091/4 "2015-12-15T13:16:23Z")

</div>

> [@Crassirostris](#):
>
> We use sync flush heavily for our cluster restart, thank you for that feature, without it, restarting our 20 TB cluster would be total pain.

Woo, glad to hear! Restarts have always been painful, we were excited to hear that synced flush is helping a lot of people. Previously we'd hear of clusters taking literally a week to perform a full rolling restart due to reshuffling shards.

> [@Crassirostris](#):
>
> Totally missed that preference too! So ashamed of myself right now, thank you a lot for clearing that out 😄 Next time I totally re-check all the documentation before asking a question.

No worries! ES is a complex piece of software, it's unrealistic for users to know all the nooks and crannies (especially stuff like `preference`, which has a pretty niche use). That's why we hang out on the forums 😄

> [@Crassirostris](#):
>
> By the way, right now ES doesn't store cluster state and that may lead to the situation, when after restart data was restored from stale replicas, because ones with the newer version didn't start yet. Are there any plans to fix it somehow or is it just the way ES is?

Yep, good question. Currently the only way to guarantee that is to have `recover_after_nodes` set to the size of your entire cluster, so you can guarantee all the existing primaries come online at the same time.

You'd may also want to block operations until the cluster health has gone to green (e.g. via a `_cluster/health?wait_for_status=green` call or similar), to ensure that all shards have started completely, although that shouldn't be a problem (if the primary is "online" but still INITIALIZING, the indexing command will block anyway).

Work is ongoing to persist the "active" shards to the cluster state, just like you suggested. That'd allow the cluster to automatically use the last active primaries, instead of potentially promoting stale replicas.

Looks like it is slated for 3.0...you can follow the progress here:

> <https://github.com/elastic/elasticsearch/issues/14739>
>
> Persist allocation IDs of active shards in cluster state and use them to recover… correct shards upon cluster restart (i.e., we can recover with only one copy and we make sure we recover the right ones and not a stale copy left around).
> 
> Relates to #14671
> \## Steps
> \- \[x\] Persist allocation ID with shard state metadata on nodes (#14831).
> \- \[x\] Persist currently started allocation IDs to index metadata (#14964).
> \- \[x\] Add allocation IDs to TransportNodesListGatewayStartedShards action. Use the above to assign a primary shard on recovery. In particular, don't assign a stale copy (#15281).
> \- \[x\] Extend reroute with an option to force assign a stale copy. This is only relevant for primaries (#15708).
> \- \[x\] Prioritize old primaries when choosing out existing \_and\_ previously active allocationIds (#16096).
> \- \[x\] Persist shard state immediately and not only on shard activation. This will help solving a race condition on primary relocation where the master has activated the target shard (removing old copy from the active set list) but the cluster crashes before the target node persisted the shard state (#16625)
> \- \[x\] Remove version from ShardRouting (#16243).

---

<div class="post-metadata">

**Author:** ![Crassirostris](https://avatars.discourse-cdn.com/v4/letter/c/779978/32.png) [@Crassirostris](https://discuss.elastic.co/u/Crassirostris)\
**Post date:** [December 15, 2015, 1:37pm UTC](https://discuss.elastic.co/t/replicas-synchronization/37091/5 "2015-12-15T13:37:10Z")

</div>

> [@polyfractal](#):
>
> No worries! ES is a complex piece of software, it's unrealistic for users to know all the nooks and crannies (especially stuff like preference, which has a pretty niche use). That's why we hang out on the forums 😄

Big thanks for that 🙂 It feels amazing to have you guys here ready to help!

> [@polyfractal](#):
>
> Work is ongoing to persist the "active" shards to the cluster state, just like you suggested. That'd allow the cluster to automatically use the last active primaries, instead of potentially promoting stale replicas.
> 
> Looks like it is slated for 3.0...you can follow the progress here:

Wow, great! Thanks for the link, sure I will follow the progress carefully. As always, great to hear that ES is getting better and better every day!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:30pm UTC](https://discuss.elastic.co/t/replicas-synchronization/37091/6 "2017-07-05T23:30:55Z")

</div>