# Replica Shard inconsistencies & disabling compression don't appear to help **URL:** \ **Category:** Elasticsearch\ **Created:** [August 21, 2014, 3:52am UTC](https://discuss.elastic.co/t/replica-shard-inconsistencies-disabling-compression-dont-appear-to-help/19379 "2014-08-21T03:52:10Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Paul\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_smith/32/1323_2.png) [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)\ **Post date:** [August 21, 2014, 3:52am UTC](https://discuss.elastic.co/t/replica-shard-inconsistencies-disabling-compression-dont-appear-to-help/19379/1 "2014-08-21T03:52:10Z")
Hi all, The recent ES 1.3.1 announcement around the ning/compression library bug had me excited because we have had a lingering replica shard inconsistency issue for a long time (on a very old ES version, which we desperately desire to upgrade, but have reasons why we can't just yet). Anyway, we've tried the Disabling Compression on the recovery setting trick and soaked the change for a few days but continue to see replica issues, and wanted to report this back. We regularly 'clean' these by shunting replica shards around to have them rebuilt, so are confident they were clean for a day, and our check tool finds small numbers still coming through each day. I heard from another ES member privately around this same change, and the 1.3.1 does fix the issue for him, but the recovery setting trick doesn't help either. I'm not sure if a full cluster start/stop is required for the recovery channel to switch to non-compressed, reading the release notes it seemed to read like a runtime setting that could be changed immediately, but it's not helping here for these types of issues. I can't yet report if ES 1.3.1 does fix the underlying issue though because of said upgrade issues (all our side, not an ES problem). anyway, thought I would pass that data point on. regards, Paul Smith -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHfYWB6FpH-i7kDs9m\_v4HgwPK-DWK75mATO\_7DZNQjpDh6B%3Dw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHfYWB6FpH-i7kDs9m_v4HgwPK-DWK75mATO_7DZNQjpDh6B%3Dw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [August 21, 2014, 8:25am UTC](https://discuss.elastic.co/t/replica-shard-inconsistencies-disabling-compression-dont-appear-to-help/19379/2 "2014-08-21T08:25:05Z")
Do you observe the replica shard inconsistency only by checksum after network transport? In other words, are you sure the inconsistency you observe is caused by a compression issue in LZF? Jörg On Thu, Aug 21, 2014 at 5:52 AM, Paul Smith [tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) wrote: > Hi all, > > The recent ES 1.3.1 announcement around the ning/compression library bug > had me excited because we have had a lingering replica shard inconsistency > issue for a long time (on a very old ES version, which we desperately > desire to upgrade, but have reasons why we can't just yet). > > Anyway, we've tried the Disabling Compression on the recovery setting > trick and soaked the change for a few days but continue to see replica > issues, and wanted to report this back. > > We regularly 'clean' these by shunting replica shards around to have them > rebuilt, so are confident they were clean for a day, and our check tool > finds small numbers still coming through each day. > > I heard from another ES member privately around this same change, and the > 1.3.1 does fix the issue for him, but the recovery setting trick doesn't > help either. > > I'm not sure if a full cluster start/stop is required for the recovery > channel to switch to non-compressed, reading the release notes it seemed to > read like a runtime setting that could be changed immediately, but it's not > helping here for these types of issues. > > I can't yet report if ES 1.3.1 does fix the underlying issue though > because of said upgrade issues (all our side, not an ES problem). > > anyway, thought I would pass that data point on. > > regards, > > Paul Smith > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAHfYWB6FpH-i7kDs9m\_v4HgwPK-DWK75mATO\_7DZNQjpDh6B%3Dw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHfYWB6FpH-i7kDs9m_v4HgwPK-DWK75mATO_7DZNQjpDh6B%3Dw%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAHfYWB6FpH-i7kDs9m\_v4HgwPK-DWK75mATO\_7DZNQjpDh6B%3Dw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHfYWB6FpH-i7kDs9m_v4HgwPK-DWK75mATO_7DZNQjpDh6B%3Dw%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEMDArRwjXw7Qv6jfD4Sa%3DiXGu-Aez6240kg3wS%3DyGT\_Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEMDArRwjXw7Qv6jfD4Sa%3DiXGu-Aez6240kg3wS%3DyGT_Q%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Paul\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_smith/32/1323_2.png) [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)\ **Post date:** [August 21, 2014, 9:21am UTC](https://discuss.elastic.co/t/replica-shard-inconsistencies-disabling-compression-dont-appear-to-help/19379/3 "2014-08-21T09:21:02Z")
No haven't looked at trying to manually compare the shard segments. Would have to manually snapshot the data (constantly under change) And no I wasn't sure our issue is related to the compression bug but it sure sounded like it. What we see is a small volume of changes relating to Deletes that do not properly delete on the replica. The records are being deleted from the primary but sometimes the replica does not seem to get this change. Interesting that the other report by another ES user how's the 1.3.1 release does fix the issue but the setting change doesn't. Since the new release is limited to mostly this library change seems restricted to this area. Only real way is for us to upgrade, which is sadly not straight forward right now. Paul On Thursday, 21 August 2014, [joergprante@gmail.com](mailto:joergprante@gmail.com) [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > Do you observe the replica shard inconsistency only by checksum after > network transport? > > In other words, are you sure the inconsistency you observe is caused by a > compression issue in LZF? > > Jörg > > On Thu, Aug 21, 2014 at 5:52 AM, Paul Smith \<[tallpsmith@gmail.com](mailto:tallpsmith@gmail.com) > \\> wrote: > > > Hi all, > > > > The recent ES 1.3.1 announcement around the ning/compression library bug > > had me excited because we have had a lingering replica shard inconsistency > > issue for a long time (on a very old ES version, which we desperately > > desire to upgrade, but have reasons why we can't just yet). > > > > Anyway, we've tried the Disabling Compression on the recovery setting > > trick and soaked the change for a few days but continue to see replica > > issues, and wanted to report this back. > > > > We regularly 'clean' these by shunting replica shards around to have them > > rebuilt, so are confident they were clean for a day, and our check tool > > finds small numbers still coming through each day. > > > > I heard from another ES member privately around this same change, and the > > 1.3.1 does fix the issue for him, but the recovery setting trick doesn't > > help either. > > > > I'm not sure if a full cluster start/stop is required for the recovery > > channel to switch to non-compressed, reading the release notes it seemed to > > read like a runtime setting that could be changed immediately, but it's not > > helping here for these types of issues. > > > > I can't yet report if ES 1.3.1 does fix the underlying issue though > > because of said upgrade issues (all our side, not an ES problem). > > > > anyway, thought I would pass that data point on. > > > > regards, > > > > Paul Smith > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) > > \ > > . > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAHfYWB6FpH-i7kDs9m\_v4HgwPK-DWK75mATO\_7DZNQjpDh6B%3Dw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHfYWB6FpH-i7kDs9m_v4HgwPK-DWK75mATO_7DZNQjpDh6B%3Dw%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAHfYWB6FpH-i7kDs9m\_v4HgwPK-DWK75mATO\_7DZNQjpDh6B%3Dw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHfYWB6FpH-i7kDs9m_v4HgwPK-DWK75mATO_7DZNQjpDh6B%3Dw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) > \ > . > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEMDArRwjXw7Qv6jfD4Sa%3DiXGu-Aez6240kg3wS%3DyGT\_Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEMDArRwjXw7Qv6jfD4Sa%3DiXGu-Aez6240kg3wS%3DyGT_Q%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEMDArRwjXw7Qv6jfD4Sa%3DiXGu-Aez6240kg3wS%3DyGT\_Q%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEMDArRwjXw7Qv6jfD4Sa%3DiXGu-Aez6240kg3wS%3DyGT_Q%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHfYWB7jQQGHn8oTN0EK80hjDQ1PjAWRDy6JWZJ\_rSdBGVsvrQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHfYWB7jQQGHn8oTN0EK80hjDQ1PjAWRDy6JWZJ_rSdBGVsvrQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:07am UTC](https://discuss.elastic.co/t/replica-shard-inconsistencies-disabling-compression-dont-appear-to-help/19379/4 "2017-07-06T01:07:15Z")
Closed on July 6, 2017, 1:07am UTC