# How to fix primary-replica inconsistency?

**URL:** <https://discuss.elastic.co/t/how-to-fix-primary-replica-inconsistency/9016>\
**Category:** Elasticsearch\
**Created:** [September 13, 2012, 9:29pm UTC](https://discuss.elastic.co/t/how-to-fix-primary-replica-inconsistency/9016 "2012-09-13T21:29:45Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![arta](https://avatars.discourse-cdn.com/v4/letter/a/aca169/32.png) [@arta](https://discuss.elastic.co/u/arta)\
**Post date:** [September 13, 2012, 9:29pm UTC](https://discuss.elastic.co/t/how-to-fix-primary-replica-inconsistency/9016/1 "2012-09-13T21:29:45Z")

</div>

Hi,  
I have a problem as same as described in here:  
[http://elasticsearch-users.115913.n3.nabble.com/BUG-Alternating-result-set-across-every-query-tt4021027.html](http://elasticsearch-users.115913.n3.nabble.com/BUG-Alternating-result-set-across-every-query-tt4021027.html)

The same search query returns one document, then next time it returns none, and alternates.  
If I add preference=\_primary\_first then I get one document every time.  
So I think the primary shard has the document but the replica does not. (I have 1 replica)

My question here is how to fix this problem.  
The discussion in the link above does not have a solution.

In addition, is there any config parameter that specifies how often or in what situation primary shards contents are reflected to their replicas?

Thanks for your help.

---

<div class="post-metadata">

**Author:** ![qjh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/qjh/32/2420_2.png) [@qjh](https://discuss.elastic.co/u/qjh)\
**Post date:** [September 14, 2012, 2:59pm UTC](https://discuss.elastic.co/t/how-to-fix-primary-replica-inconsistency/9016/2 "2012-09-14T14:59:28Z")

</div>

I have been seeing this same issue on a few indices. The primary and  
replica are divergent and nothing seems to resolve it (they have been  
refreshed and optimized). I've since worked around this by having to  
recreate the index.

Does anyone have a good cluster state dump that could by used to open an  
issue? There doesn't appear to be one for this yet:

> **[Issues · elastic/elasticsearch](https://github.com/elastic/elasticsearch/issues)**
>
> Free and Open, Distributed, RESTful Search Engine. Contribute to elastic/elasticsearch development by creating an account on GitHub.

On Thursday, September 13, 2012 5:29:48 PM UTC-4, arta wrote:

> Hi,  
> I have a problem as same as described in here:
> 
> [http://elasticsearch-users.115913.n3.nabble.com/BUG-Alternating-result-set-across-every-query-tt4021027.html](http://elasticsearch-users.115913.n3.nabble.com/BUG-Alternating-result-set-across-every-query-tt4021027.html)
> 
> The same search query returns one document, then next time it returns  
> none,  
> and alternates.  
> If I add preference=\_primary\_first then I get one document every time.  
> So I think the primary shard has the document but the replica does not. (I  
> have 1 replica)
> 
> My question here is how to fix this problem.  
> The discussion in the link above does not have a solution.
> 
> In addition, is there any config parameter that specifies how often or in  
> what situation primary shards contents are reflected to their replicas?
> 
> Thanks for your help.
> 
> --  
> View this message in context:  
> [http://elasticsearch-users.115913.n3.nabble.com/How-to-fix-primary-replica-inconsistency-tp4022692.html](http://elasticsearch-users.115913.n3.nabble.com/How-to-fix-primary-replica-inconsistency-tp4022692.html)  
> Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com).

--

---

<div class="post-metadata">

**Author:** ![Kurt\_Harriger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kurt_harriger/32/2676_2.png) [@Kurt\_Harriger](https://discuss.elastic.co/u/Kurt_Harriger)\
**Post date:** [September 17, 2012, 4:39pm UTC](https://discuss.elastic.co/t/how-to-fix-primary-replica-inconsistency/9016/3 "2012-09-17T16:39:10Z")

</div>

Same issue here and same resolution at the moment. I just haven't had the  
time yet to dig deep into this issue, but its something we will need to dig  
into soon. At the moment we're limping along with a  
?preference=\_primary\_first to prevent users from seeing the  
inconsistencies, so our replica shard basically only acts as a hot standby  
in case the master fails. We already had code that writes to  
multiple indices simultaneously that we used to migrate from solr and/or  
make schema changes, so we just use this code to keep two indices up to  
date. This way when we discover an issue with one index we switch to the  
other index drop the broken one and rebuild it without affecting our users.

We currently use the \_status endpoint to identify if the replicas our out  
of sync, I tried to write a script to do this and force an index failover  
and reindex automatically but I wasn't how to determine if the numDocs  
differed because the replica shard hasn't yet applied all the change sets  
or if it was actually out of sync.

On Friday, September 14, 2012 8:59:29 AM UTC-6, qjh wrote:

> I have been seeing this same issue on a few indices. The primary and  
> replica are divergent and nothing seems to resolve it (they have been  
> refreshed and optimized). I've since worked around this by having to  
> recreate the index.
> 
> Does anyone have a good cluster state dump that could by used to open an  
> issue? There doesn't appear to be one for this yet:  
> [Issues · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues)
> 
> On Thursday, September 13, 2012 5:29:48 PM UTC-4, arta wrote:
> 
> > Hi,  
> > I have a problem as same as described in here:
> > 
> > [http://elasticsearch-users.115913.n3.nabble.com/BUG-Alternating-result-set-across-every-query-tt4021027.html](http://elasticsearch-users.115913.n3.nabble.com/BUG-Alternating-result-set-across-every-query-tt4021027.html)
> > 
> > The same search query returns one document, then next time it returns  
> > none,  
> > and alternates.  
> > If I add preference=\_primary\_first then I get one document every time.  
> > So I think the primary shard has the document but the replica does not.  
> > (I  
> > have 1 replica)
> > 
> > My question here is how to fix this problem.  
> > The discussion in the link above does not have a solution.
> > 
> > In addition, is there any config parameter that specifies how often or in  
> > what situation primary shards contents are reflected to their replicas?
> > 
> > Thanks for your help.
> > 
> > --  
> > View this message in context:  
> > [http://elasticsearch-users.115913.n3.nabble.com/How-to-fix-primary-replica-inconsistency-tp4022692.html](http://elasticsearch-users.115913.n3.nabble.com/How-to-fix-primary-replica-inconsistency-tp4022692.html)  
> > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com).

--

---

<div class="post-metadata">

**Author:** ![arta](https://avatars.discourse-cdn.com/v4/letter/a/aca169/32.png) [@arta](https://discuss.elastic.co/u/arta)\
**Post date:** [September 17, 2012, 6:08pm UTC](https://discuss.elastic.co/t/how-to-fix-primary-replica-inconsistency/9016/4 "2012-09-17T18:08:42Z")

</div>

Thanks for the reply, qjh, Kurt.  
Seems like there is no way other than reindexing to fix the problem.

Kurt mentioned about numDocs, I suppose that means we can use curl with \_count to determine whether there is inconsistency between shards.

I have millions of documents.  
If there is a way to find out which documents are only in primary or only in replica, that information makes the reindexing a lot efficient.

---

<div class="post-metadata">

**Author:** ![arta](https://avatars.discourse-cdn.com/v4/letter/a/aca169/32.png) [@arta](https://discuss.elastic.co/u/arta)\
**Post date:** [September 19, 2012, 5:34pm UTC](https://discuss.elastic.co/t/how-to-fix-primary-replica-inconsistency/9016/5 "2012-09-19T17:34:09Z")

</div>

Kurt,  
Can you please elaborate "We currently use the \_status endpoint to identify if the replicas our out of sync" a little more?  
So you compare each shard's num\_docs?

As you mentioned, while indexing is running on, it will be difficult to to distinguish the cause of the number difference, i.e. by inconsistency or by replication delay.  
What is your strategy to automatically discover the inconsistency?

Thanks again for your help!

---

<div class="post-metadata">

**Author:** ![Kurt\_Harriger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kurt_harriger/32/2676_2.png) [@Kurt\_Harriger](https://discuss.elastic.co/u/Kurt_Harriger)\
**Post date:** [September 19, 2012, 8:10pm UTC](https://discuss.elastic.co/t/how-to-fix-primary-replica-inconsistency/9016/6 "2012-09-19T20:10:45Z")

</div>

Yep, I basically just look to see how different the num\_docs is. There is  
also a translog/id which I would assume that if master and replica shard  
have the same translog/id then they should have the same num\_docs. I was  
thinking perhaps writing a script that looks for any shards that have same  
translog/id but different num\_docs, but for now I just check it manually.  
In any event here is a typical response I get back that appears out of  
sync to me.

{

- ok: true,
- \_shards:  
{
  - total: 20,
  - successful: 20,
  - failed: 0  
},

- indices:  
{
  - default2:  
{
    - index:  
{
      - primary\_size: "3.5gb",
      - primary\_size\_in\_bytes: 3836722124,
      - size: "7gb",
      - size\_in\_bytes: 7606933703  
},

    - translog:  
{
      - operations: 1621  
},

    - docs:  
{
      - num\_docs: 3065297,
      - max\_doc: 3462692,
      - deleted\_docs: 397395  
},

    - merges:  
{
      - current: 0,
      - current\_docs: 0,
      - current\_size: "0b",
      - current\_size\_in\_bytes: 0,
      - total: 43004,
      - total\_time: "3h",
      - total\_time\_in\_millis: 10957007,
      - total\_docs: 71491119,
      - total\_size: "79.9gb",
      - total\_size\_in\_bytes: 85824354912  
},

    - refresh:  
{
      - total: 226327,
      - total\_time: "1.1h",
      - total\_time\_in\_millis: 4198337  
},

    - flush:  
{
      - total: 3828,
      - total\_time: "59.9m",
      - total\_time\_in\_millis: 3596409  
},

    - shards:  
{
      - 
## 0: [

## { - routing: { - state: "STARTED", - primary: true, - node: "95gp\_xKVRra472UxiDygiA", - relocating\_node: null, - shard: 0, - index: "default2" }, - state: "STARTED", - index: { - size: "373.5mb", - size\_in\_bytes: 391702523 }, - translog: { - id: 1347901831292, - operations: 74 }, - docs: { - num\_docs: 307291, - max\_doc: 353087, - deleted\_docs: 45796 }, - merges: { - current: 0, - current\_docs: 0, - current\_size: "0b", - current\_size\_in\_bytes: 0, - total: 2143, - total\_time: "12.8m", - total\_time\_in\_millis: 770853, - total\_docs: 3179382, - total\_size: "3.5gb", - total\_size\_in\_bytes: 3843614285 }, - refresh: { - total: 11204, - total\_time: "5.2m", - total\_time\_in\_millis: 317926 }, - flush: { - total: 192, - total\_time: "4.7m", - total\_time\_in\_millis: 283969 } },
{  
- routing:  
{  
- state: "STARTED",  
- primary: false,  
- node: "tdhHCSEBSaKLmsaE\_E4Gzw",  
- relocating\_node: null,  
- shard: 0,  
- index: "default2"  
},  
- state: "STARTED",  
- index:  
{  
- size: "362.3mb",  
- size\_in\_bytes: 379899395  
},  
- translog:  
{  
- id: 1347901831292,  
- operations: 74  
},  
- docs:  
{  
- num\_docs: 303379,  
- max\_doc: 343611,  
- deleted\_docs: 40232  
},  
- merges:  
{  
- current: 0,  
- current\_docs: 0,  
- current\_size: "0b",  
- current\_size\_in\_bytes: 0,  
- total: 2173,  
- total\_time: "5.4m",  
- total\_time\_in\_millis: 329683,  
- total\_docs: 3825448,  
- total\_size: "4.2gb",  
- total\_size\_in\_bytes: 4573205157  
},  
- refresh:  
{  
- total: 11583,  
- total\_time: "1.7m",  
- total\_time\_in\_millis: 107880  
},  
- flush:  
{  
- total: 192,  
- total\_time: "1.7m",  
- total\_time\_in\_millis: 107054  
}  
}  
],
      - 
## 1: [

## { - routing: { - state: "STARTED", - primary: true, - node: "95gp\_xKVRra472UxiDygiA", - relocating\_node: null, - shard: 1, - index: "default2" }, - state: "STARTED", - index: { - size: "363.6mb", - size\_in\_bytes: 381298886 }, - translog: { - id: 1347901831415, - operations: 96 }, - docs: { - num\_docs: 306295, - max\_doc: 344589, - deleted\_docs: 38294 }, - merges: { - current: 0, - current\_docs: 0, - current\_size: "0b", - current\_size\_in\_bytes: 0, - total: 2148, - total\_time: "13.5m", - total\_time\_in\_millis: 815476, - total\_docs: 3661059, - total\_size: "4.1gb", - total\_size\_in\_bytes: 4447223751 }, - refresh: { - total: 11265, - total\_time: "5.2m", - total\_time\_in\_millis: 312026 }, - flush: { - total: 191, - total\_time: "4.3m", - total\_time\_in\_millis: 258892 } },
{  
- routing:  
{  
- state: "STARTED",  
- primary: false,  
- node: "tdhHCSEBSaKLmsaE\_E4Gzw",  
- relocating\_node: null,  
- shard: 1,  
- index: "default2"  
},  
- state: "STARTED",  
- index:  
{  
- size: "378.3mb",  
- size\_in\_bytes: 396779935  
},  
- translog:  
{  
- id: 1347901831415,  
- operations: 96  
},  
- docs:  
{  
- num\_docs: 302098,  
- max\_doc: 356028,  
- deleted\_docs: 53930  
},  
- merges:  
{  
- current: 0,  
- current\_docs: 0,  
- current\_size: "0b",  
- current\_size\_in\_bytes: 0,  
- total: 2168,  
- total\_time: "4.9m",  
- total\_time\_in\_millis: 297319,  
- total\_docs: 3164854,  
- total\_size: "3.5gb",  
- total\_size\_in\_bytes: 3803854038  
},  
- refresh:  
{  
- total: 11656,  
- total\_time: "1.7m",  
- total\_time\_in\_millis: 107995  
},  
- flush:  
{  
- total: 191,  
- total\_time: "1.8m",  
- total\_time\_in\_millis: 109000  
}  
}  
],
      - 
## 2: [

## { - routing: { - state: "STARTED", - primary: false, - node: "95gp\_xKVRra472UxiDygiA", - relocating\_node: null, - shard: 2, - index: "default2" }, - state: "STARTED", - index: { - size: "341.9mb", - size\_in\_bytes: 358525806 }, - translog: { - id: 1347901772812, - operations: 90 }, - docs: { - num\_docs: 305291, - max\_doc: 328203, - deleted\_docs: 22912 }, - merges: { - current: 0, - current\_docs: 0, - current\_size: "0b", - current\_size\_in\_bytes: 0, - total: 2138, - total\_time: "12.3m", - total\_time\_in\_millis: 741442, - total\_docs: 3817398, - total\_size: "4.2gb", - total\_size\_in\_bytes: 4560351643 }, - refresh: { - total: 11297, - total\_time: "5.1m", - total\_time\_in\_millis: 311758 }, - flush: { - total: 191, - total\_time: "3.4m", - total\_time\_in\_millis: 208984 } },
{  
- routing:  
{  
- state: "STARTED",  
- primary: true,  
- node: "tdhHCSEBSaKLmsaE\_E4Gzw",  
- relocating\_node: null,  
- shard: 2,  
- index: "default2"  
},  
- state: "STARTED",  
- index:  
{  
- size: "355.3mb",  
- size\_in\_bytes: 372645152  
},  
- translog:  
{  
- id: 1347901772813,  
- operations: 90  
},  
- docs:  
{  
- num\_docs: 306395,  
- max\_doc: 339658,  
- deleted\_docs: 33263  
},  
- merges:  
{  
- current: 0,  
- current\_docs: 0,  
- current\_size: "0b",  
- current\_size\_in\_bytes: 0,  
- total: 2186,  
- total\_time: "5.1m",  
- total\_time\_in\_millis: 310026,  
- total\_docs: 3408416,  
- total\_size: "3.8gb",  
- total\_size\_in\_bytes: 4086555557  
},  
- refresh:  
{  
- total: 11587,  
- total\_time: "1.7m",  
- total\_time\_in\_millis: 102844  
},  
- flush:  
{  
- total: 192,  
- total\_time: "1.6m",  
- total\_time\_in\_millis: 96582  
}  
}  
],
      - 
## 3: [

## { - routing: { - state: "STARTED", - primary: false, - node: "95gp\_xKVRra472UxiDygiA", - relocating\_node: null, - shard: 3, - index: "default2" }, - state: "STARTED", - index: { - size: "401.3mb", - size\_in\_bytes: 420864740 }, - translog: { - id: 1347901772798, - operations: 67 }, - docs: { - num\_docs: 305251, - max\_doc: 375754, - deleted\_docs: 70503 }, - merges: { - current: 0, - current\_docs: 0, - current\_size: "0b", - current\_size\_in\_bytes: 0, - total: 2124, - total\_time: "12.4m", - total\_time\_in\_millis: 747821, - total\_docs: 3515161, - total\_size: "3.9gb", - total\_size\_in\_bytes: 4238118231 }, - refresh: { - total: 11146, - total\_time: "5.2m", - total\_time\_in\_millis: 317066 }, - flush: { - total: 191, - total\_time: "3.9m", - total\_time\_in\_millis: 235077 } },
{  
- routing:  
{  
- state: "STARTED",  
- primary: true,  
- node: "tdhHCSEBSaKLmsaE\_E4Gzw",  
- relocating\_node: null,  
- shard: 3,  
- index: "default2"  
},  
- state: "STARTED",  
- index:  
{  
- size: "404.1mb",  
- size\_in\_bytes: 423736704  
},  
- translog:  
{  
- id: 1347901772799,  
- operations: 69  
},  
- docs:  
{  
- num\_docs: 306550,  
- max\_doc: 377493,  
- deleted\_docs: 70943  
},  
- merges:  
{  
- current: 0,  
- current\_docs: 0,  
- current\_size: "0b",  
- current\_size\_in\_bytes: 0,  
- total: 2175,  
- total\_time: "5.2m",  
- total\_time\_in\_millis: 316727,  
- total\_docs: 3449317,  
- total\_size: "3.8gb",  
- total\_size\_in\_bytes: 4152370061  
},  
- refresh:  
{  
- total: 11438,  
- total\_time: "1.8m",  
- total\_time\_in\_millis: 109234  
},  
- flush:  
{  
- total: 192,  
- total\_time: "2.1m",  
- total\_time\_in\_millis: 126846  
}  
}  
],
      - 
## 4: [

## { - routing: { - state: "STARTED", - primary: true, - node: "95gp\_xKVRra472UxiDygiA", - relocating\_node: null, - shard: 4, - index: "default2" }, - state: "STARTED", - index: { - size: "381.6mb", - size\_in\_bytes: 400202939 }, - translog: { - id: 1347901831289, - operations: 98 }, - docs: { - num\_docs: 305897, - max\_doc: 359662, - deleted\_docs: 53765 }, - merges: { - current: 0, - current\_docs: 0, - current\_size: "0b", - current\_size\_in\_bytes: 0, - total: 2138, - total\_time: "12.4m", - total\_time\_in\_millis: 748422, - total\_docs: 3747681, - total\_size: "4.1gb", - total\_size\_in\_bytes: 4482284498 }, - refresh: { - total: 11228, - total\_time: "5.3m", - total\_time\_in\_millis: 319209 }, - flush: { - total: 190, - total\_time: "4m", - total\_time\_in\_millis: 243675 } },
{  
- routing:  
{  
- state: "STARTED",  
- primary: false,  
- node: "tdhHCSEBSaKLmsaE\_E4Gzw",  
- relocating\_node: null,  
- shard: 4,  
- index: "default2"  
},  
- state: "STARTED",  
- index:  
{  
- size: "350.3mb",  
- size\_in\_bytes: 367367654  
},  
- translog:  
{  
- id: 1347901831290,  
- operations: 97  
},  
- docs:  
{  
- num\_docs: 302085,  
- max\_doc: 331367,  
- deleted\_docs: 29282  
},  
- merges:  
{  
- current: 0,  
- current\_docs: 0,  
- current\_size: "0b",  
- current\_size\_in\_bytes: 0,  
- total: 2160,  
- total\_time: "5.4m",  
- total\_time\_in\_millis: 325400,  
- total\_docs: 3663609,  
- total\_size: "4gb",  
- total\_size\_in\_bytes: 4395019498  
},  
- refresh:  
{  
- total: 11551,  
- total\_time: "1.8m",  
- total\_time\_in\_millis: 108807  
},  
- flush:  
{  
- total: 191,  
- total\_time: "2.1m",  
- total\_time\_in\_millis: 128884  
}  
}  
],
      - 
## 5: [

## { - routing: { - state: "STARTED", - primary: true, - node: "95gp\_xKVRra472UxiDygiA", - relocating\_node: null, - shard: 5, - index: "default2" }, - state: "STARTED", - index: { - size: "362.2mb", - size\_in\_bytes: 379848618 }, - translog: { - id: 1347901831423, - operations: 82 }, - docs: { - num\_docs: 306784, - max\_doc: 340679, - deleted\_docs: 33895 }, - merges: { - current: 0, - current\_docs: 0, - current\_size: "0b", - current\_size\_in\_bytes: 0, - total: 2125, - total\_time: "13.2m", - total\_time\_in\_millis: 795503, - total\_docs: 3530900, - total\_size: "3.9gb", - total\_size\_in\_bytes: 4257253724 }, - refresh: { - total: 11088, - total\_time: "5m", - total\_time\_in\_millis: 302650 }, - flush: { - total: 192, - total\_time: "4.6m", - total\_time\_in\_millis: 277611 } },
{  
- routing:  
{  
- state: "STARTED",  
- primary: false,  
- node: "tdhHCSEBSaKLmsaE\_E4Gzw",  
- relocating\_node: null,  
- shard: 5,  
- index: "default2"  
},  
- state: "STARTED",  
- index:  
{  
- size: "344.1mb",  
- size\_in\_bytes: 360850766  
},  
- translog:  
{  
- id: 1347901831423,  
- operations: 80  
},  
- docs:  
{  
- num\_docs: 303711,  
- max\_doc: 325543,  
- deleted\_docs: 21832  
},  
- merges:  
{  
- current: 0,  
- current\_docs: 0,  
- current\_size: "0b",  
- current\_size\_in\_bytes: 0,  
- total: 2154,  
- total\_time: "5.6m",  
- total\_time\_in\_millis: 337964,  
- total\_docs: 3825832,  
- total\_size: "4.2gb",  
- total\_size\_in\_bytes: 4591861598  
},  
- refresh:  
{  
- total: 11433,  
- total\_time: "1.7m",  
- total\_time\_in\_millis: 104688  
},  
- flush:  
{  
- total: 192,  
- total\_time: "1.5m",  
- total\_time\_in\_millis: 91405  
}  
}  
],
      - 
## 6: [

## { - routing: { - state: "STARTED", - primary: true, - node: "95gp\_xKVRra472UxiDygiA", - relocating\_node: null, - shard: 6, - index: "default2" }, - state: "STARTED", - index: { - size: "361.9mb", - size\_in\_bytes: 379558406 }, - translog: { - id: 1347901831378, - operations: 54 }, - docs: { - num\_docs: 306499, - max\_doc: 343743, - deleted\_docs: 37244 }, - merges: { - current: 0, - current\_docs: 0, - current\_size: "0b", - current\_size\_in\_bytes: 0, - total: 2128, - total\_time: "13.2m", - total\_time\_in\_millis: 796938, - total\_docs: 3601219, - total\_size: "4gb", - total\_size\_in\_bytes: 4305769584 }, - refresh: { - total: 11016, - total\_time: "5m", - total\_time\_in\_millis: 303480 }, - flush: { - total: 192, - total\_time: "4.7m", - total\_time\_in\_millis: 286890 } },
{  
- routing:  
{  
- state: "STARTED",  
- primary: false,  
- node: "tdhHCSEBSaKLmsaE\_E4Gzw",  
- relocating\_node: null,  
- shard: 6,  
- index: "default2"  
},  
- state: "STARTED",  
- index:  
{  
- size: "391.3mb",  
- size\_in\_bytes: 410343610  
},  
- translog:  
{  
- id: 1347901831378,  
- operations: 54  
},  
- docs:  
{  
- num\_docs: 302809,  
- max\_doc: 366089,  
- deleted\_docs: 63280  
},  
- merges:  
{  
- current: 0,  
- current\_docs: 0,  
- current\_size: "0b",  
- current\_size\_in\_bytes: 0,  
- total: 2144,  
- total\_time: "5.4m",  
- total\_time\_in\_millis: 324305,  
- total\_docs: 3584721,  
- total\_size: "4gb",  
- total\_size\_in\_bytes: 4300272618  
},  
- refresh:  
{  
- total: 11354,  
- total\_time: "1.8m",  
- total\_time\_in\_millis: 112237  
},  
- flush:  
{  
- total: 192,  
- total\_time: "1.9m",  
- total\_time\_in\_millis: 115561  
}  
}  
],
      - 
## 7: [

## { - routing: { - state: "STARTED", - primary: true, - node: "95gp\_xKVRra472UxiDygiA", - relocating\_node: null, - shard: 7, - index: "default2" }, - state: "STARTED", - index: { - size: "358.4mb", - size\_in\_bytes: 375896103 }, - translog: { - id: 1347901831301, - operations: 74 }, - docs: { - num\_docs: 306144, - max\_doc: 337143, - deleted\_docs: 30999 }, - merges: { - current: 0, - current\_docs: 0, - current\_size: "0b", - current\_size\_in\_bytes: 0, - total: 2125, - total\_time: "13.2m", - total\_time\_in\_millis: 796752, - total\_docs: 3564015, - total\_size: "3.9gb", - total\_size\_in\_bytes: 4286994130 }, - refresh: { - total: 11048, - total\_time: "5.2m", - total\_time\_in\_millis: 316874 }, - flush: { - total: 191, - total\_time: "3.9m", - total\_time\_in\_millis: 236884 } },
{  
- routing:  
{  
- state: "STARTED",  
- primary: false,  
- node: "tdhHCSEBSaKLmsaE\_E4Gzw",  
- relocating\_node: null,  
- shard: 7,  
- index: "default2"  
},  
- state: "STARTED",  
- index:  
{  
- size: "344.2mb",  
- size\_in\_bytes: 360935272  
},  
- translog:  
{  
- id: 1347901831301,  
- operations: 73  
},  
- docs:  
{  
- num\_docs: 302416,  
- max\_doc: 327545,  
- deleted\_docs: 25129  
},  
- merges:  
{  
- current: 0,  
- current\_docs: 0,  
- current\_size: "0b",  
- current\_size\_in\_bytes: 0,  
- total: 2163,  
- total\_time: "5.6m",  
- total\_time\_in\_millis: 336263,  
- total\_docs: 3810258,  
- total\_size: "4.2gb",  
- total\_size\_in\_bytes: 4565132999  
},  
- refresh:  
{  
- total: 11416,  
- total\_time: "1.7m",  
- total\_time\_in\_millis: 105342  
},  
- flush:  
{  
- total: 191,  
- total\_time: "1.3m",  
- total\_time\_in\_millis: 80964  
}  
}  
],
      - 
## 8: [

## { - routing: { - state: "STARTED", - primary: false, - node: "95gp\_xKVRra472UxiDygiA", - relocating\_node: null, - shard: 8, - index: "default2" }, - state: "STARTED", - index: { - size: "330.7mb", - size\_in\_bytes: 346790325 }, - translog: { - id: 1347901772804, - operations: 66 }, - docs: { - num\_docs: 305785, - max\_doc: 315564, - deleted\_docs: 9779 }, - merges: { - current: 0, - current\_docs: 0, - current\_size: "0b", - current\_size\_in\_bytes: 0, - total: 2116, - total\_time: "13.2m", - total\_time\_in\_millis: 796557, - total\_docs: 3818340, - total\_size: "4.2gb", - total\_size\_in\_bytes: 4556889217 }, - refresh: { - total: 11042, - total\_time: "5m", - total\_time\_in\_millis: 305252 }, - flush: { - total: 191, - total\_time: "3.5m", - total\_time\_in\_millis: 215837 } },
{  
- routing:  
{  
- state: "STARTED",  
- primary: true,  
- node: "tdhHCSEBSaKLmsaE\_E4Gzw",  
- relocating\_node: null,  
- shard: 8,  
- index: "default2"  
},  
- state: "STARTED",  
- index:  
{  
- size: "349.5mb",  
- size\_in\_bytes: 366535393  
},  
- translog:  
{  
- id: 1347901772805,  
- operations: 64  
},  
- docs:  
{  
- num\_docs: 306854,  
- max\_doc: 334557,  
- deleted\_docs: 27703  
},  
- merges:  
{  
- current: 0,  
- current\_docs: 0,  
- current\_size: "0b",  
- current\_size\_in\_bytes: 0,  
- total: 2175,  
- total\_time: "4.9m",  
- total\_time\_in\_millis: 299631,  
- total\_docs: 3358427,  
- total\_size: "3.7gb",  
- total\_size\_in\_bytes: 4019011835  
},  
- refresh:  
{  
- total: 11362,  
- total\_time: "1.7m",  
- total\_time\_in\_millis: 102257  
},  
- flush:  
{  
- total: 192,  
- total\_time: "1.8m",  
- total\_time\_in\_millis: 109720  
}  
}  
],
      - 
## 9: [

## { - routing: { - state: "STARTED", - primary: false, - node: "95gp\_xKVRra472UxiDygiA", - relocating\_node: null, - shard: 9, - index: "default2" }, - state: "STARTED", - index: { - size: "350.8mb", - size\_in\_bytes: 367854076 }, - translog: { - id: 1347901772820, - operations: 111 }, - docs: { - num\_docs: 305293, - max\_doc: 333894, - deleted\_docs: 28601 }, - merges: { - current: 0, - current\_docs: 0, - current\_size: "0b", - current\_size\_in\_bytes: 0, - total: 2130, - total\_time: "12.3m", - total\_time\_in\_millis: 740824, - total\_docs: 3148931, - total\_size: "3.5gb", - total\_size\_in\_bytes: 3773382780 }, - refresh: { - total: 11139, - total\_time: "5.5m", - total\_time\_in\_millis: 330083 }, - flush: { - total: 191, - total\_time: "4.1m", - total\_time\_in\_millis: 247113 } },
{  
- routing:  
{  
- state: "STARTED",  
- primary: true,  
- node: "tdhHCSEBSaKLmsaE\_E4Gzw",  
- relocating\_node: null,  
- shard: 9,  
- index: "default2"  
},  
- state: "STARTED",  
- index:  
{  
- size: "348.3mb",  
- size\_in\_bytes: 365297400  
},  
- translog:  
{  
- id: 1347901772820,  
- operations: 112  
},  
- docs:  
{  
- num\_docs: 306588,  
- max\_doc: 332081,  
- deleted\_docs: 25493  
},  
- merges:  
{  
- current: 0,  
- current\_docs: 0,  
- current\_size: "0b",  
- current\_size\_in\_bytes: 0,  
- total: 2191,  
- total\_time: "5.4m",  
- total\_time\_in\_millis: 329101,  
- total\_docs: 3816151,  
- total\_size: "4.2gb",  
- total\_size\_in\_bytes: 4585189708  
},  
- refresh:  
{  
- total: 11474,  
- total\_time: "1.6m",  
- total\_time\_in\_millis: 100729  
},  
- flush:  
{  
- total: 191,  
- total\_time: "2.2m",  
- total\_time\_in\_millis: 135461  
}  
}  
]  
}  
}  
}

}

--

---

<div class="post-metadata">

**Author:** ![arta](https://avatars.discourse-cdn.com/v4/letter/a/aca169/32.png) [@arta](https://discuss.elastic.co/u/arta)\
**Post date:** [September 20, 2012, 5:05pm UTC](https://discuss.elastic.co/t/how-to-fix-primary-replica-inconsistency/9016/7 "2012-09-20T17:05:33Z")

</div>

Thanks, again, Kurt,  
I looked at \_status output and I think what I found was inconsistency disappears over time.  
I wrote a simple ruby script to process \_status?pretty=true output (see below)  
I took a couple of sample data and saw many shards have different translog id but the num\_docs seem ok.  
What does translog id actually mean?  
What is your concern if replicas have different translog id?

One more question for anybody knows ES well:  
Some document seemed to take hours until it appeared in the replica (or maybe viceversa).  
What brings this time lag?

----------------------------------------- here's my script --------------  
if (ARGV.length \< 1)  
puts $PROGRAM\_NAME + ' '  
exit  
end

@mode = :waiting\_index  
@info = {}

open(ARGV[0], 'r') {|f|  
f.each\_line {|line|  
case @mode  
when :waiting\_index  
if (line =~ /"(i[0-9]+)" : /)  
@index = $1  
@info[@index] = {}  
@mode = :waiting\_shard\_no  
end  
when :waiting\_shard\_no  
if (line =~ /"([0-9]+)" : [/)  
@shard = $1  
@info[@index][@shard] = {}  
@mode = :handling\_shard  
@subshard = 0  
@info[@index][@shard][@subshard] = {}  
end  
when :handling\_shard  
if (line =~ /"primary" : (\w+),/)  
@info[@index][@shard][@subshard][:primary] = $1  
elsif (line =~ /"node" : "(\S+)",/)  
@info[@index][@shard][@subshard][:node] = $1  
elsif (line =~ /"id" : ([0-9]+)/)  
@info[@index][@shard][@subshard][:translog\_id] = $1  
elsif (line =~ /"num\_docs" : ([0-9]+)/)  
@info[@index][@shard][@subshard][:num\_docs] = $1  
elsif (line =~ /^\s+}, {\s?$/)  
@subshard += 1  
@info[@index][@shard][@subshard] = {}  
elsif (line =~ /^\s+} ],\s?$/)  
@mode = :waiting\_shard\_no  
elsif (line =~ /^\s+} ]\s?$/)  
@mode = :waiting\_index  
end  
end  
}  
}

def dump\_info(info)  
"tranalog=#{info[:translog\_id]} num\_docs=#{info[:num\_docs]} node=#{info[:node]}" +  
(info[:primary] == "true" ? " primary" : " ")  
end

def dump\_diff(idx, shard, i, info, ref\_i, ref\_info)  
idxShard = "#{idx}-#{shard}"  
"#{idxShard} [#{ref\_i}]: #{dump\_info(ref\_info)}\n" +  
" " \* idxShard.length + " [#{i}]: #{dump\_info(info)}"  
end

def sort\_keys(col)  
col.keys.sort {|a,b|  
if (a.class != String || a.length == b.length)  
a \<=\> b  
else  
a.length \<=\> b.length  
end  
}  
end

sort\_keys(@info).each {|idx| idx\_info = @info[idx]  
sort\_keys(idx\_info).each {|shard| shard\_info = idx\_info[shard]  
translogs = []  
sort\_keys(shard\_info).each {|i| info = shard\_info[i]  
if (translogs.empty?)  
translogs \<\< { i =\> info }  
else  
if (info[:num\_docs] != translogs[0].values[0][:num\_docs])  
puts dump\_diff(idx, shard, i, info, translogs[0].keys[0], translogs[0].values[0]) + " DIFFERENT COUNT"  
elsif (info[:translog\_id] != translogs[0].values[0][:translog\_id])  
puts dump\_diff(idx, shard, i, info, translogs[0].keys[0], translogs[0].values[0])  
end  
end  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![Kurt\_Harriger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kurt_harriger/32/2676_2.png) [@Kurt\_Harriger](https://discuss.elastic.co/u/Kurt_Harriger)\
**Post date:** [September 20, 2012, 5:33pm UTC](https://discuss.elastic.co/t/how-to-fix-primary-replica-inconsistency/9016/8 "2012-09-20T17:33:54Z")

</div>

I assume that the translog/id represents the position in the transaction log. If the replica does not have the same id then I would assume it is still catching up to the master and one would expect that the num\_docs may differ as the replica is still replaying changes. However, if they are at the same position in the transaction log and the num\_docs is different then I would assume that the replica should have the same num\_docs as the master and if not then why not… that is the question. I still haven't had any time to dig into the Elasticsearch source code so that is just my assumptions, perhaps a committer could clarify.

--  
Kurt Harriger  
Sent with Sparrow ([http://www.sparrowmailapp.com/?sig](http://www.sparrowmailapp.com/?sig))

On Thursday, September 20, 2012 at 11:05 AM, arta wrote:

> Thanks, again, Kurt,  
> I looked at \_status output and I think what I found was inconsistency  
> disappears over time.  
> I wrote a simple ruby script to process \_status?pretty=true output (see  
> below)  
> I took a couple of sample data and saw many shards have different translog  
> id but the num\_docs seem ok.  
> What does translog id actually mean?  
> What is your concern if replicas have different translog id?
> 
> One more question for anybody knows ES well:  
> Some document seemed to take hours until it appeared in the replica (or  
> maybe viceversa).  
> What brings this time lag?
> 
> ----------------------------------------- here's my script --------------  
> if (ARGV.length \< 1)  
> puts $PROGRAM\_NAME + ' '  
> exit  
> end
> 
> @mode = :waiting\_index  
> @info = {}
> 
> open(ARGV[0], 'r') {|f|  
> f.each\_line {|line|  
> case @mode  
> when :waiting\_index  
> if (line =~ /"(i[0-9]+)" : /)  
> @index = $1  
> @info[@index] = {}  
> @mode = :waiting\_shard\_no  
> end  
> when :waiting\_shard\_no  
> if (line =~ /"([0-9]+)" : [/)  
> @shard = $1  
> @info[@index][@shard] = {}  
> @mode = :handling\_shard  
> @subshard = 0  
> @info[@index][@shard][@subshard] = {}  
> end  
> when :handling\_shard  
> if (line =~ /"primary" : (\w+),/)  
> @info[@index][@shard][@subshard][:primary] = $1  
> elsif (line =~ /"node" : "(\S+)",/)  
> @info[@index][@shard][@subshard][:node] = $1  
> elsif (line =~ /"id" : ([0-9]+)/)  
> @info[@index][@shard][@subshard][:translog\_id] = $1  
> elsif (line =~ /"num\_docs" : ([0-9]+)/)  
> @info[@index][@shard][@subshard][:num\_docs] = $1  
> elsif (line =~ /^\s+}, {\s?$/)  
> @subshard += 1  
> @info[@index][@shard][@subshard] = {}  
> elsif (line =~ /^\s+} ],\s?$/)  
> @mode = :waiting\_shard\_no  
> elsif (line =~ /^\s+} ]\s?$/)  
> @mode = :waiting\_index  
> end  
> end  
> }  
> }
> 
> def dump\_info(info)  
> "tranalog=#{info[:translog\_id]} num\_docs=#{info[:num\_docs]}  
> node=#{info[:node]}" +  
> (info[:primary] == "true" ? " primary" : " ")  
> end
> 
> def dump\_diff(idx, shard, i, info, ref\_i, ref\_info)  
> idxShard = "#{idx}-#{shard}"  
> "#{idxShard} [#{ref\_i}]: #{dump\_info(ref\_info)}\n" +  
> " " \* idxShard.length + " [#{i}]: #{dump\_info(info)}"  
> end
> 
> def sort\_keys(col)  
> col.keys.sort {|a,b|  
> if (a.class != String || a.length == b.length)  
> a \<=\> b  
> else  
> a.length \<=\> b.length  
> end  
> }  
> end
> 
> sort\_keys(@info).each {|idx| idx\_info = @info[idx]  
> sort\_keys(idx\_info).each {|shard| shard\_info = idx\_info[shard]  
> translogs =   
> sort\_keys(shard\_info).each {|i| info = shard\_info[i]  
> if (translogs.empty?)  
> translogs \<\< { i =\> info }  
> else  
> if (info[:num\_docs] != translogs[0].values[0][:num\_docs])  
> puts dump\_diff(idx, shard, i, info, translogs[0].keys[0],  
> translogs[0].values[0]) + " DIFFERENT COUNT"  
> elsif (info[:translog\_id] != translogs[0].values[0][:translog\_id])  
> puts dump\_diff(idx, shard, i, info, translogs[0].keys[0],  
> translogs[0].values[0])  
> end  
> end  
> }  
> }  
> }
> 
> --  
> View this message in context: [http://elasticsearch-users.115913.n3.nabble.com/How-to-fix-primary-replica-inconsistency-tp4022692p4022926.html](http://elasticsearch-users.115913.n3.nabble.com/How-to-fix-primary-replica-inconsistency-tp4022692p4022926.html)  
> Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com) ([http://Nabble.com](http://Nabble.com)).
> 
> --

--

---

<div class="post-metadata">

**Author:** ![es\_learner](https://avatars.discourse-cdn.com/v4/letter/e/8dc957/32.png) [@es\_learner](https://discuss.elastic.co/u/es_learner)\
**Post date:** [October 18, 2012, 5:46am UTC](https://discuss.elastic.co/t/how-to-fix-primary-replica-inconsistency/9016/9 "2012-10-18T05:46:31Z")

</div>

May I know which version of ES you are using? I'm on 0.19.2 and have been hitting primary only. BUT lately, because of increased traffic causing high CPU spikes, I am planning to load-balance reads across my 5 replicas, 3 servers cluster. Reading this thread gives me pause. Any new info will help me greatly.

Thanks.

curl localhost:9200/?version

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [October 18, 2012, 10:51am UTC](https://discuss.elastic.co/t/how-to-fix-primary-replica-inconsistency/9016/10 "2012-10-18T10:51:11Z")

</div>

The translation id does not indicate "consistency" between shards, its  
internal to each shard. Which version are you using? Also, anything  
interesting in the logs (failures)?

On Thursday, September 20, 2012 7:34:00 PM UTC+2, Kurt Harriger wrote:

> I assume that the translog/id represents the position in the transaction  
> log. If the replica does not have the same id then I would assume it is  
> still catching up to the master and one would expect that the num\_docs may  
> differ as the replica is still replaying changes. However, if they are at  
> the same position in the transaction log and the num\_docs is different then  
> I would assume that the replica should have the same num\_docs as the master  
> and if not then why not… that is the question. I still haven't had any  
> time to dig into the Elasticsearch source code so that is just my  
> assumptions, perhaps a committer could clarify.
> 
> --  
> Kurt Harriger  
> Sent with Sparrow [http://www.sparrowmailapp.com/?sig](http://www.sparrowmailapp.com/?sig)
> 
> On Thursday, September 20, 2012 at 11:05 AM, arta wrote:
> 
> Thanks, again, Kurt,  
> I looked at \_status output and I think what I found was inconsistency  
> disappears over time.  
> I wrote a simple ruby script to process \_status?pretty=true output (see  
> below)  
> I took a couple of sample data and saw many shards have different translog  
> id but the num\_docs seem ok.  
> What does translog id actually mean?  
> What is your concern if replicas have different translog id?
> 
> One more question for anybody knows ES well:  
> Some document seemed to take hours until it appeared in the replica (or  
> maybe viceversa).  
> What brings this time lag?
> 
> ----------------------------------------- here's my script --------------  
> if (ARGV.length \< 1)  
> puts $PROGRAM\_NAME + ' '  
> exit  
> end
> 
> @mode = :waiting\_index  
> @info = {}
> 
> open(ARGV[0], 'r') {|f|  
> f.each\_line {|line|  
> case @mode  
> when :waiting\_index  
> if (line =~ /"(i[0-9]+)" : /)  
> @index = $1  
> @info[@index] = {}  
> @mode = :waiting\_shard\_no  
> end  
> when :waiting\_shard\_no  
> if (line =~ /"([0-9]+)" : [/)  
> @shard = $1  
> @info[@index][@shard] = {}  
> @mode = :handling\_shard  
> @subshard = 0  
> @info[@index][@shard][@subshard] = {}  
> end  
> when :handling\_shard  
> if (line =~ /"primary" : (\w+),/)  
> @info[@index][@shard][@subshard][:primary] = $1  
> elsif (line =~ /"node" : "(\S+)",/)  
> @info[@index][@shard][@subshard][:node] = $1  
> elsif (line =~ /"id" : ([0-9]+)/)  
> @info[@index][@shard][@subshard][:translog\_id] = $1  
> elsif (line =~ /"num\_docs" : ([0-9]+)/)  
> @info[@index][@shard][@subshard][:num\_docs] = $1  
> elsif (line =~ /^\s+}, {\s?$/)  
> @subshard += 1  
> @info[@index][@shard][@subshard] = {}  
> elsif (line =~ /^\s+} ],\s?$/)  
> @mode = :waiting\_shard\_no  
> elsif (line =~ /^\s+} ]\s?$/)  
> @mode = :waiting\_index  
> end  
> end  
> }  
> }
> 
> def dump\_info(info)  
> "tranalog=#{info[:translog\_id]} num\_docs=#{info[:num\_docs]}  
> node=#{info[:node]}" +  
> (info[:primary] == "true" ? " primary" : " ")  
> end
> 
> def dump\_diff(idx, shard, i, info, ref\_i, ref\_info)  
> idxShard = "#{idx}-#{shard}"  
> "#{idxShard} [#{ref\_i}]: #{dump\_info(ref\_info)}\n" +  
> " " \* idxShard.length + " [#{i}]: #{dump\_info(info)}"  
> end
> 
> def sort\_keys(col)  
> col.keys.sort {|a,b|  
> if (a.class != String || a.length == b.length)  
> a \<=\> b  
> else  
> a.length \<=\> b.length  
> end  
> }  
> end
> 
> sort\_keys(@info).each {|idx| idx\_info = @info[idx]  
> sort\_keys(idx\_info).each {|shard| shard\_info = idx\_info[shard]  
> translogs =   
> sort\_keys(shard\_info).each {|i| info = shard\_info[i]  
> if (translogs.empty?)  
> translogs \<\< { i =\> info }  
> else  
> if (info[:num\_docs] != translogs[0].values[0][:num\_docs])  
> puts dump\_diff(idx, shard, i, info, translogs[0].keys[0],  
> translogs[0].values[0]) + " DIFFERENT COUNT"  
> elsif (info[:translog\_id] != translogs[0].values[0][:translog\_id])  
> puts dump\_diff(idx, shard, i, info, translogs[0].keys[0],  
> translogs[0].values[0])  
> end  
> end  
> }  
> }  
> }
> 
> --  
> View this message in context:  
> [http://elasticsearch-users.115913.n3.nabble.com/How-to-fix-primary-replica-inconsistency-tp4022692p4022926.html](http://elasticsearch-users.115913.n3.nabble.com/How-to-fix-primary-replica-inconsistency-tp4022692p4022926.html)  
> Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com).
> 
> --

--

---

<div class="post-metadata">

**Author:** ![arta](https://avatars.discourse-cdn.com/v4/letter/a/aca169/32.png) [@arta](https://discuss.elastic.co/u/arta)\
**Post date:** [October 18, 2012, 5:34pm UTC](https://discuss.elastic.co/t/how-to-fix-primary-replica-inconsistency/9016/11 "2012-10-18T17:34:10Z")

</div>

In my case, 0.19.3 it is.  
I found the discrepancy disappeared after a while.  
In some case it took days, but most of cases within an hour.  
I don't see any related entry in logs.

--

---

<div class="post-metadata">

**Author:** ![Kurt\_Harriger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kurt_harriger/32/2676_2.png) [@Kurt\_Harriger](https://discuss.elastic.co/u/Kurt_Harriger)\
**Post date:** [October 18, 2012, 8:40pm UTC](https://discuss.elastic.co/t/how-to-fix-primary-replica-inconsistency/9016/12 "2012-10-18T20:40:34Z")

</div>

Using version 0.19.8 here. I haven't looked much deeper into the issue since adding preference=\_primary\_first. Without primary\_first the hit count would change on nearly every query. Given that same transaction id does not indicate that the replica is caught up with the master its possible that the issue might resolve itself given enough time, however when we first encountered the hit counts alternating between values the issue persisted for two days on test accounts so I don't think it is likely that time alone would have resolved it.

How does one determine if the shards are inconsistent or just behind? I don't know, but it would be nice if there was a way to get a more definitive answer to this question.

--  
Kurt Harriger  
Sent with Sparrow ([http://www.sparrowmailapp.com/?sig](http://www.sparrowmailapp.com/?sig))

On Thursday, October 18, 2012 at 11:34 AM, arta wrote:

> ## In my case, 0.19.3 it is. I found the discrepancy disappeared after a while. In some case it took days, but most of cases within an hour. I don't see any related entry in logs.

--

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [October 19, 2012, 10:37am UTC](https://discuss.elastic.co/t/how-to-fix-primary-replica-inconsistency/9016/13 "2012-10-19T10:37:36Z")

</div>

> How does one determine if the shards are inconsistent or just behind?  
> I don't know, but it would be nice if there was a way to get a more  
> definitive answer to this question.

The shards should be neither inconsistent nor behind. The one exception  
might be where you use 'async' indexing.

You can use the cluster\_health API (with eg level=shards) to get a view  
of your cluster.

clint

> 

--

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [October 20, 2012, 11:00pm UTC](https://discuss.elastic.co/t/how-to-fix-primary-replica-inconsistency/9016/14 "2012-10-20T23:00:52Z")

</div>

One more thing, make sure not to confuse primary-replica inconsistency with "counts" being different. Let me explain, but first, note that 0.19.5 and later 0.19.7 fixed bugs that might resolve in inconsistencies (though under quite extreme cases, i.e. only managed to recreate it in a test that ran for 5 days with nodes constantly being killed).

Back to the "inconsistency" part, if you keep on indexing into an index, obviously, you will see some "inconsistencies" between calls as data keeps being added to the index. Also, those changes will be visible as shards will be refreshed (by default, 1s by default).

Also, when executing a search, with sorting, for example, based on \_score (the default), some docs will have the same \_score, and there isn't consistency there (unless using additional sort field). So, when executing it "once" and then another time, it might hit other shard copies, and sorting there based on same \_score value would be different.

Note, the above problems, even though both shards have the same data, it might seem like they are giving different results.

How do you solve it? The simplest way that I personally like is to use the preference option, but using a dynamic value. If you do preference=[user\_id] (for example), then for the same user id, the same shard copies will be hit.

On Oct 19, 2012, at 12:37 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) wrote:

> > How does one determine if the shards are inconsistent or just behind?  
> > I don't know, but it would be nice if there was a way to get a more  
> > definitive answer to this question.
> 
> The shards should be neither inconsistent nor behind. The one exception  
> might be where you use 'async' indexing.
> 
> You can use the cluster\_health API (with eg level=shards) to get a view  
> of your cluster.
> 
> clint
> 
> > 
> 
> --

--

---

<div class="post-metadata">

**Author:** ![Filirom1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/filirom1/32/2234_2.png) [@Filirom1](https://discuss.elastic.co/u/Filirom1)\
**Post date:** [November 16, 2012, 9:18am UTC](https://discuss.elastic.co/t/how-to-fix-primary-replica-inconsistency/9016/15 "2012-11-16T09:18:20Z")

</div>

I confirm the issue on Elasticsearch 0.19.11 .

With the bulk API, I reindex all my data. Now the indexation has finished  
but the total number of hits is different on primary and replica :

?preference=\_primary\_first  
hits.total: 12209124

?preference=\_replica\_first  
hits.total: 12209202

I think this problem is the same as this one :

> <https://github.com/elastic/elasticsearch/issues/2354>
>
> Hi, 
> 
> On an empty 1 node cluster elasticsearch v0.19.10, I want to reproduce a M…apperParsingException\[object mapping for \[my\_type\] tried to parse as object, but got EOF, has a concrete value been provided to it?\]
> 
> But it's not so easy, here is the script I loop on:
> 
> \`\`\`
> curl -s -XDELETE http://localhost:9200/my\_index/?pretty=true;
> 
> curl -s -XPUT 'http://localhost:9200/\_bulk?pretty=true' --data-binary '
> {"index":{"\_index":"my\_index","\_type":"my\_type"}}
> {"obj":{"key":"value"}}
> {"index":{"\_index":"my\_index","\_type":"my\_type"}}
> {"obj":"value"}
> '
> \`\`\`
> 
> Sometimes I have the Exception, and sometimes not : 
> 
> \`\`\`
> \[root@dahu share\]# curl -s -XDELETE http://localhost:9200/my\_index/?pretty=true;
> 
> curl -s -XPUT 'http://localhost:9200/\_bulk?pretty=true' --data-binary '
> {"index":{"\_index":"my\_index","\_type":"my\_type"}}
> {"obj":{"key":"value"}}
> {"index":{"\_index":"my\_index","\_type":"my\_type"}}
> {"obj":"value"}
> '{
> "ok" : true,
> "acknowledged" : true
> }\[root@dahu share\]#
> \[root@dahu share\]# curl -s -XPUT 'http://localhost:9200/\_bulk?pretty=true' --data-binary '
> \> {"index":{"\_index":"my\_index","\_type":"my\_type"}}
> \> {"obj":{"key":"value"}}
> \> {"index":{"\_index":"my\_index","\_type":"my\_type"}}
> \> {"obj":"value"}
> \> '
> {
> "took" : 221,
> "items" : \[ {
> "create" : {
> "\_index" : "my\_index",
> "\_type" : "my\_type",
> "\_id" : "QJz1\_TNWT9yhf7Mu5cRlDw",
> "\_version" : 1,
> "ok" : true
> }
> }, {
> "create" : {
> "\_index" : "my\_index",
> "\_type" : "my\_type",
> "\_id" : "ibUrM7JzRaGfalbgaF-aTA",
> "error" : "MapperParsingException\[object mapping for \[my\_type\] tried to parse as object, but got EOF, has a concrete value been provided to it?\]"
> }
> } \]
> }\[root@dahu share\]#
> 
> 
> \[root@dahu share\]# curl -s -XDELETE http://localhost:9200/my\_index/?pretty=true;
> curl -s -XPUT 'http://localhost:9200/\_bulk?pretty=true' --data-binary '
> {"index":{"\_index":"my\_index","\_type":"my\_type"}}
> {"obj":{"key":"value"}}
> {"index":{"\_index":"my\_index","\_type":"my\_type"}}
> {"obj":"value"}
> '{
> "ok" : true,
> "acknowledged" : true
> }\[root@dahu share\]#
> \[root@dahu share\]# curl -s -XPUT 'http://localhost:9200/\_bulk?pretty=true' --data-binary '
> \> {"index":{"\_index":"my\_index","\_type":"my\_type"}}
> \> {"obj":{"key":"value"}}
> \> {"index":{"\_index":"my\_index","\_type":"my\_type"}}
> \> {"obj":"value"}
> \> '
> {
> "took" : 251,
> "items" : \[ {
> "create" : {
> "\_index" : "my\_index",
> "\_type" : "my\_type",
> "\_id" : "N2PS-hoPQv2mLN7wzkZFHg",
> "\_version" : 1,
> "ok" : true
> }
> }, {
> "create" : {
> "\_index" : "my\_index",
> "\_type" : "my\_type",
> "\_id" : "-G9gf7kYRTi3Jb2A4-OsVQ",
> "\_version" : 1,
> "ok" : true
> }
> } \]
> \`\`\`
> 
> Quite strange...

When I try to index the same bulk of documents (on an empty index)  
sometimes I have an error and sometimes not.

I think this is the cause of inconsitency between primary and replica.

Cheers  
Romain

2012/10/21 [kimchy@gmail.com](mailto:kimchy@gmail.com)

> One more thing, make sure not to confuse primary-replica inconsistency  
> with "counts" being different. Let me explain, but first, note that 0.19.5  
> and later 0.19.7 fixed bugs that might resolve in inconsistencies (though  
> under quite extreme cases, i.e. only managed to recreate it in a test that  
> ran for 5 days with nodes constantly being killed).
> 
> Back to the "inconsistency" part, if you keep on indexing into an index,  
> obviously, you will see some "inconsistencies" between calls as data keeps  
> being added to the index. Also, those changes will be visible as shards  
> will be refreshed (by default, 1s by default).
> 
> Also, when executing a search, with sorting, for example, based on \_score  
> (the default), some docs will have the same \_score, and there isn't  
> consistency there (unless using additional sort field). So, when executing  
> it "once" and then another time, it might hit other shard copies, and  
> sorting there based on same \_score value would be different.
> 
> Note, the above problems, even though both shards have the same data, it  
> might seem like they are giving different results.
> 
> How do you solve it? The simplest way that I personally like is to use the  
> preference option, but using a dynamic value. If you do  
> preference=[user\_id] (for example), then for the same user id, the same  
> shard copies will be hit.
> 
> On Oct 19, 2012, at 12:37 PM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com)  
> wrote:
> 
> > > How does one determine if the shards are inconsistent or just behind?  
> > > I don't know, but it would be nice if there was a way to get a more  
> > > definitive answer to this question.
> > 
> > The shards should be neither inconsistent nor behind. The one exception  
> > might be where you use 'async' indexing.
> > 
> > You can use the cluster\_health API (with eg level=shards) to get a view  
> > of your cluster.
> > 
> > clint
> > 
> > > 
> > 
> > --
> 
> --

--

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [November 16, 2012, 4:54pm UTC](https://discuss.elastic.co/t/how-to-fix-primary-replica-inconsistency/9016/16 "2012-11-16T16:54:49Z")

</div>

Hi Filirom1,

if you see mapping exceptions, something in your JSON data style is  
inconsistent, see my comment

> <https://github.com/elastic/elasticsearch/issues/2354>

Jörg

--

---

<div class="post-metadata">

**Author:** ![Filirom1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/filirom1/32/2234_2.png) [@Filirom1](https://discuss.elastic.co/u/Filirom1)\
**Post date:** [November 16, 2012, 5:20pm UTC](https://discuss.elastic.co/t/how-to-fix-primary-replica-inconsistency/9016/17 "2012-11-16T17:20:50Z")

</div>

Yes I know, but I can't change what the users inject in Elasticsearch.

The point is that sometimes an inconsistent JSON is accepted by ES.

2012/11/16 Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com)

> Hi Filirom1,
> 
> if you see mapping exceptions, something in your JSON data style is  
> inconsistent, see my comment
> 
> [Sometimes MapperParsingException and sometimes not · Issue #2354 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2354#issuecomment-10453428)
> 
> Jörg
> 
> --

--

---

<div class="post-metadata">

**Author:** ![Tal\_Shemesh](https://avatars.discourse-cdn.com/v4/letter/t/7cd45c/32.png) [@Tal\_Shemesh](https://discuss.elastic.co/u/Tal_Shemesh)\
**Post date:** [April 13, 2014, 6:41am UTC](https://discuss.elastic.co/t/how-to-fix-primary-replica-inconsistency/9016/18 "2014-04-13T06:41:59Z")

</div>

Hi,

we are facing the same issue with 0.90.11.  
we have a shard that it's primary size is 1.9gb and the replica is 1gb.  
did you manage to solve the problem?  
if so, how can we fix it?

On Friday, November 16, 2012 7:21:16 PM UTC+2, Filirom1 wrote:

> Yes I know, but I can't change what the users inject in Elasticsearch.
> 
> The point is that sometimes an inconsistent JSON is accepted by ES.
> 
> 2012/11/16 Jörg Prante \<[joerg...@gmail.com](mailto:joerg...@gmail.com) \<javascript:\>\>
> 
> > Hi Filirom1,
> > 
> > if you see mapping exceptions, something in your JSON data style is  
> > inconsistent, see my comment
> > 
> > [Sometimes MapperParsingException and sometimes not · Issue #2354 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2354#issuecomment-10453428)
> > 
> > Jörg
> > 
> > --

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/86314b57-e292-491e-94ac-92a5a35b8344%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/86314b57-e292-491e-94ac-92a5a35b8344%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\
**Post date:** [April 14, 2014, 9:02am UTC](https://discuss.elastic.co/t/how-to-fix-primary-replica-inconsistency/9016/19 "2014-04-14T09:02:56Z")

</div>

Hi,

A difference in disk size doesn't mean that they don't have the same  
content since one of the replicas might just have run a large merge that  
saved disk space. Nevertheless, you can force shards to be re-replicated by  
using the update setting API[1] to temporarily set the number of replicas  
to 0 (this will deallocate replicas) and then back to the original value  
(which will cause replicas to be bulk-copied from the primaries).

[1]

> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.

On Sun, Apr 13, 2014 at 8:41 AM, Tal Shemesh [tal.shemesh@gmail.com](mailto:tal.shemesh@gmail.com) wrote:

> Hi,
> 
> we are facing the same issue with 0.90.11.  
> we have a shard that it's primary size is 1.9gb and the replica is 1gb.  
> did you manage to solve the problem?  
> if so, how can we fix it?
> 
> On Friday, November 16, 2012 7:21:16 PM UTC+2, Filirom1 wrote:
> 
> > Yes I know, but I can't change what the users inject in Elasticsearch.
> > 
> > The point is that sometimes an inconsistent JSON is accepted by ES.
> > 
> > 2012/11/16 Jörg Prante [joerg...@gmail.com](mailto:joerg...@gmail.com)
> > 
> > > Hi Filirom1,
> > > 
> > > if you see mapping exceptions, something in your JSON data style is  
> > > inconsistent, see my comment  
> > > [Sometimes MapperParsingException and sometimes not · Issue #2354 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2354#issuecomment-)  
> > > 10453428
> > > 
> > > Jörg
> > > 
> > > --
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/86314b57-e292-491e-94ac-92a5a35b8344%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/86314b57-e292-491e-94ac-92a5a35b8344%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/86314b57-e292-491e-94ac-92a5a35b8344%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/86314b57-e292-491e-94ac-92a5a35b8344%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .  
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
Adrien Grand

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6ofT81ApiPMnx%2BFvogfp\_sHQyqKWMmTStDWFq%2B5Cvttg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6ofT81ApiPMnx%2BFvogfp_sHQyqKWMmTStDWFq%2B5Cvttg%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:36am UTC](https://discuss.elastic.co/t/how-to-fix-primary-replica-inconsistency/9016/20 "2017-07-06T01:36:04Z")

</div>