# Inconsistent data among nodes for one (or more) indexes (ES 0.11) **URL:** \ **Category:** Elasticsearch\ **Created:** [October 8, 2010, 7:06pm UTC](https://discuss.elastic.co/t/inconsistent-data-among-nodes-for-one-or-more-indexes-es-0-11/3426 "2010-10-08T19:06:09Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Pablo\_Borges](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pablo_borges/32/3313_2.png) [@Pablo\_Borges](https://discuss.elastic.co/u/Pablo_Borges)\ **Post date:** [October 8, 2010, 7:06pm UTC](https://discuss.elastic.co/t/inconsistent-data-among-nodes-for-one-or-more-indexes-es-0-11/3426/1 "2010-10-08T19:06:09Z")
Here's one of the problems I got frequently. I have a 5 node ES (using local storage) cluster running 8 indexes each with thousands of documents (one of them with millions) and every now and then it seems like replication breaks: for node in n1 n2 n3 n4 n5; do curl "http://${node}:9200/index/\_count?q=\*" {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} The only solution found so far is to recreate the index and reindex. Any help will be _really_ appreciatted! 😃 ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 8, 2010, 7:17pm UTC](https://discuss.elastic.co/t/inconsistent-data-among-nodes-for-one-or-more-indexes-es-0-11/3426/2 "2010-10-08T19:17:51Z")
Have you seen the problem with recovery and local gateway with more than one index I posted? Also, do you see any exceptions in the log? The index seems to have just one shard, how many replicas does it have? -shay.banon On Fri, Oct 8, 2010 at 9:06 PM, Pablo Borges [pablort@gmail.com](mailto:pablort@gmail.com) wrote: > Here's one of the problems I got frequently. > > I have a 5 node ES (using local storage) cluster running 8 indexes each > with thousands of documents (one of them with millions) and every now and > then it seems like replication breaks: > > for node in n1 n2 n3 n4 n5; do > curl "http://${node}:9200/index/\_count?q=\*" > > {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} > {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > > The only solution found so far is to recreate the index and reindex. > > Any help will be _really_ appreciatted! 😃 ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 8, 2010, 7:23pm UTC](https://discuss.elastic.co/t/inconsistent-data-among-nodes-for-one-or-more-indexes-es-0-11/3426/3 "2010-10-08T19:23:42Z")
Also, a few more points on the count: 1. Going to different nodes is irrelevant. The count will round robin between the shards and be directed to the appropriate node(s). 2. Do you index while you execute the count? Have you changed the refresh interval? -shay.banon On Fri, Oct 8, 2010 at 9:17 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Have you seen the problem with recovery and local gateway with more than > one index I posted? Also, do you see any exceptions in the log? The index > seems to have just one shard, how many replicas does it have? > > -shay.banon > > On Fri, Oct 8, 2010 at 9:06 PM, Pablo Borges [pablort@gmail.com](mailto:pablort@gmail.com) wrote: > > > Here's one of the problems I got frequently. > > > > I have a 5 node ES (using local storage) cluster running 8 indexes each > > with thousands of documents (one of them with millions) and every now and > > then it seems like replication breaks: > > > > for node in n1 n2 n3 n4 n5; do > > curl "http://${node}:9200/index/\_count?q=\*" > > > > {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} > > {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} > > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > > > > The only solution found so far is to recreate the index and reindex. > > > > Any help will be _really_ appreciatted! 😃 ---
**Author:** ![Pablo\_Borges](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pablo_borges/32/3313_2.png) [@Pablo\_Borges](https://discuss.elastic.co/u/Pablo_Borges)\ **Post date:** [October 8, 2010, 8:04pm UTC](https://discuss.elastic.co/t/inconsistent-data-among-nodes-for-one-or-more-indexes-es-0-11/3426/4 "2010-10-08T20:04:20Z")
About 1. If you notice, I use only one shard per node, which means one of them (al least) is bogged. About 2. No. The data has been indexed sometime ago and updates are not frequent. On Fri, Oct 8, 2010 at 4:23 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Also, a few more points on the count: > > 1. Going to different nodes is irrelevant. The count will round robin > between the shards and be directed to the appropriate node(s). > > 2. Do you index while you execute the count? Have you changed the refresh > interval? > > -shay.banon > > On Fri, Oct 8, 2010 at 9:17 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > > Have you seen the problem with recovery and local gateway with more than > > one index I posted? Also, do you see any exceptions in the log? The index > > seems to have just one shard, how many replicas does it have? > > > > -shay.banon > > > > On Fri, Oct 8, 2010 at 9:06 PM, Pablo Borges [pablort@gmail.com](mailto:pablort@gmail.com) wrote: > > > > > Here's one of the problems I got frequently. > > > > > > I have a 5 node ES (using local storage) cluster running 8 indexes each > > > with thousands of documents (one of them with millions) and every now and > > > then it seems like replication breaks: > > > > > > for node in n1 n2 n3 n4 n5; do > > > curl "http://${node}:9200/index/\_count?q=\*" > > > > > > {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} > > > {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} > > > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > > > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > > > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > > > > > > The only solution found so far is to recreate the index and reindex. > > > > > > Any help will be _really_ appreciatted! 😃 ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 8, 2010, 8:05pm UTC](https://discuss.elastic.co/t/inconsistent-data-among-nodes-for-one-or-more-indexes-es-0-11/3426/5 "2010-10-08T20:05:58Z")
But do they still happen? On Fri, Oct 8, 2010 at 10:04 PM, Pablo Borges [pablort@gmail.com](mailto:pablort@gmail.com) wrote: > About 1. If you notice, I use only one shard per node, which means one of > them (al least) is bogged. > About 2. No. The data has been indexed sometime ago and updates are not > frequent. > > On Fri, Oct 8, 2010 at 4:23 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > > Also, a few more points on the count: > > > > 1. Going to different nodes is irrelevant. The count will round robin > > between the shards and be directed to the appropriate node(s). > > > > 2. Do you index while you execute the count? Have you changed the refresh > > interval? > > > > -shay.banon > > > > On Fri, Oct 8, 2010 at 9:17 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > > > > Have you seen the problem with recovery and local gateway with more than > > > one index I posted? Also, do you see any exceptions in the log? The index > > > seems to have just one shard, how many replicas does it have? > > > > > > -shay.banon > > > > > > On Fri, Oct 8, 2010 at 9:06 PM, Pablo Borges [pablort@gmail.com](mailto:pablort@gmail.com) wrote: > > > > > > > Here's one of the problems I got frequently. > > > > > > > > I have a 5 node ES (using local storage) cluster running 8 indexes each > > > > with thousands of documents (one of them with millions) and every now and > > > > then it seems like replication breaks: > > > > > > > > for node in n1 n2 n3 n4 n5; do > > > > curl "http://${node}:9200/index/\_count?q=\*" > > > > > > > > {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} > > > > {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} > > > > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > > > > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > > > > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > > > > > > > > The only solution found so far is to recreate the index and reindex. > > > > > > > > Any help will be _really_ appreciatted! 😃 ---
**Author:** ![Pablo\_Borges](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pablo_borges/32/3313_2.png) [@Pablo\_Borges](https://discuss.elastic.co/u/Pablo_Borges)\ **Post date:** [October 8, 2010, 8:06pm UTC](https://discuss.elastic.co/t/inconsistent-data-among-nodes-for-one-or-more-indexes-es-0-11/3426/6 "2010-10-08T20:06:02Z")
No exceptions, but I needed to reduce my log level in order not to fill the partition the logs are written (several search exceptions, which I'd like to know if there's a way to disable) Here's my logging.yml rootLogger: INFO, console, file logger: # log action execution errors for easier debugging action : INFO appender: console: type: console layout: type: consolePattern conversionPattern: "[%d{ABSOLUTE}][%-5p][%-25c] %m%n" file: type: dailyRollingFile file: /var/log/elasticsearch/${cluster.name}.log datePattern: "'.'yyyy-MM-dd" layout: type: pattern conversionPattern: "[%d{ABSOLUTE}][%-5p][%-25c] %m%n" On Fri, Oct 8, 2010 at 4:17 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Have you seen the problem with recovery and local gateway with more than > one index I posted? Also, do you see any exceptions in the log? The index > seems to have just one shard, how many replicas does it have? > > -shay.banon > > On Fri, Oct 8, 2010 at 9:06 PM, Pablo Borges [pablort@gmail.com](mailto:pablort@gmail.com) wrote: > > > Here's one of the problems I got frequently. > > > > I have a 5 node ES (using local storage) cluster running 8 indexes each > > with thousands of documents (one of them with millions) and every now and > > then it seems like replication breaks: > > > > for node in n1 n2 n3 n4 n5; do > > curl "http://${node}:9200/index/\_count?q=\*" > > > > {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} > > {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} > > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > > > > The only solution found so far is to recreate the index and reindex. > > > > Any help will be _really_ appreciatted! 😃 ---
**Author:** ![Pablo\_Borges](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pablo_borges/32/3313_2.png) [@Pablo\_Borges](https://discuss.elastic.co/u/Pablo_Borges)\ **Post date:** [October 8, 2010, 8:07pm UTC](https://discuss.elastic.co/t/inconsistent-data-among-nodes-for-one-or-more-indexes-es-0-11/3426/7 "2010-10-08T20:07:00Z")
Yes: {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} On Fri, Oct 8, 2010 at 5:05 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > But do they still happen? > > On Fri, Oct 8, 2010 at 10:04 PM, Pablo Borges [pablort@gmail.com](mailto:pablort@gmail.com) wrote: > > > About 1. If you notice, I use only one shard per node, which means one of > > them (al least) is bogged. > > About 2. No. The data has been indexed sometime ago and updates are not > > frequent. > > > > On Fri, Oct 8, 2010 at 4:23 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > > > > Also, a few more points on the count: > > > > > > 1. Going to different nodes is irrelevant. The count will round robin > > > between the shards and be directed to the appropriate node(s). > > > > > > 2. Do you index while you execute the count? Have you changed the refresh > > > interval? > > > > > > -shay.banon > > > > > > On Fri, Oct 8, 2010 at 9:17 PM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) > > > > > > > wrote: > > > > > > > Have you seen the problem with recovery and local gateway with more than > > > > one index I posted? Also, do you see any exceptions in the log? The index > > > > seems to have just one shard, how many replicas does it have? > > > > > > > > -shay.banon > > > > > > > > On Fri, Oct 8, 2010 at 9:06 PM, Pablo Borges [pablort@gmail.com](mailto:pablort@gmail.com) wrote: > > > > > > > > > Here's one of the problems I got frequently. > > > > > > > > > > I have a 5 node ES (using local storage) cluster running 8 indexes each > > > > > with thousands of documents (one of them with millions) and every now and > > > > > then it seems like replication breaks: > > > > > > > > > > for node in n1 n2 n3 n4 n5; do > > > > > curl "http://${node}:9200/index/\_count?q=\*" > > > > > > > > > > {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} > > > > > {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} > > > > > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > > > > > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > > > > > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > > > > > > > > > > The only solution found so far is to recreate the index and reindex. > > > > > > > > > > Any help will be _really_ appreciatted! 😃 ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 8, 2010, 8:13pm UTC](https://discuss.elastic.co/t/inconsistent-data-among-nodes-for-one-or-more-indexes-es-0-11/3426/8 "2010-10-08T20:13:53Z")
I suggest you keep action on DEBUG, I can help with the search exceptions as well. On Fri, Oct 8, 2010 at 10:06 PM, Pablo Borges [pablort@gmail.com](mailto:pablort@gmail.com) wrote: > No exceptions, but I needed to reduce my log level in order not to fill the > partition the logs are written (several search exceptions, which I'd like to > know if there's a way to disable) > > Here's my logging.yml > > rootLogger: INFO, console, file > logger: > > # log action execution errors for easier debugging > > action : INFO > > appender: > console: > type: console > layout: > type: consolePattern > conversionPattern: "[%d{ABSOLUTE}][%-5p][%-25c] %m%n" > > file: > type: dailyRollingFile > file: /var/log/elasticsearch/${cluster.name}.log > datePattern: "'.'yyyy-MM-dd" > layout: > type: pattern > conversionPattern: "[%d{ABSOLUTE}][%-5p][%-25c] %m%n" > > On Fri, Oct 8, 2010 at 4:17 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > > Have you seen the problem with recovery and local gateway with more than > > one index I posted? Also, do you see any exceptions in the log? The index > > seems to have just one shard, how many replicas does it have? > > > > -shay.banon > > > > On Fri, Oct 8, 2010 at 9:06 PM, Pablo Borges [pablort@gmail.com](mailto:pablort@gmail.com) wrote: > > > > > Here's one of the problems I got frequently. > > > > > > I have a 5 node ES (using local storage) cluster running 8 indexes each > > > with thousands of documents (one of them with millions) and every now and > > > then it seems like replication breaks: > > > > > > for node in n1 n2 n3 n4 n5; do > > > curl "http://${node}:9200/index/\_count?q=\*" > > > > > > {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} > > > {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} > > > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > > > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > > > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > > > > > > The only solution found so far is to recreate the index and reindex. > > > > > > Any help will be _really_ appreciatted! 😃 ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 8, 2010, 8:15pm UTC](https://discuss.elastic.co/t/inconsistent-data-among-nodes-for-one-or-more-indexes-es-0-11/3426/9 "2010-10-08T20:15:03Z")
IF they still happen, you might get into the near real time aspect of elasticsearch ... . The question is if they are not the same once there are no indexing going on. Also, note that there is a problem with multiple indices and the local gateway. -shay.banon On Fri, Oct 8, 2010 at 10:07 PM, Pablo Borges [pablort@gmail.com](mailto:pablort@gmail.com) wrote: > Yes: > > {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} > {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} > > On Fri, Oct 8, 2010 at 5:05 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > > But do they still happen? > > > > On Fri, Oct 8, 2010 at 10:04 PM, Pablo Borges [pablort@gmail.com](mailto:pablort@gmail.com) wrote: > > > > > About 1. If you notice, I use only one shard per node, which means one of > > > them (al least) is bogged. > > > About 2. No. The data has been indexed sometime ago and updates are not > > > frequent. > > > > > > On Fri, Oct 8, 2010 at 4:23 PM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) > > > > > > > wrote: > > > > > > > Also, a few more points on the count: > > > > > > > > 1. Going to different nodes is irrelevant. The count will round robin > > > > between the shards and be directed to the appropriate node(s). > > > > > > > > 2. Do you index while you execute the count? Have you changed the > > > > refresh interval? > > > > > > > > -shay.banon > > > > > > > > On Fri, Oct 8, 2010 at 9:17 PM, Shay Banon \< > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote: > > > > > > > > > Have you seen the problem with recovery and local gateway with more > > > > > than one index I posted? Also, do you see any exceptions in the log? The > > > > > index seems to have just one shard, how many replicas does it have? > > > > > > > > > > -shay.banon > > > > > > > > > > On Fri, Oct 8, 2010 at 9:06 PM, Pablo Borges [pablort@gmail.com](mailto:pablort@gmail.com)wrote: > > > > > > > > > > > Here's one of the problems I got frequently. > > > > > > > > > > > > I have a 5 node ES (using local storage) cluster running 8 indexes > > > > > > each with thousands of documents (one of them with millions) and every now > > > > > > and then it seems like replication breaks: > > > > > > > > > > > > for node in n1 n2 n3 n4 n5; do > > > > > > curl "http://${node}:9200/index/\_count?q=\*" > > > > > > > > > > > > {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} > > > > > > {"count":22408,"\_shards":{"total":1,"successful":1,"failed":0}} > > > > > > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > > > > > > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > > > > > > {"count":20230,"\_shards":{"total":1,"successful":1,"failed":0}} > > > > > > > > > > > > The only solution found so far is to recreate the index and reindex. > > > > > > > > > > > > Any help will be _really_ appreciatted! 😃 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:18am UTC](https://discuss.elastic.co/t/inconsistent-data-among-nodes-for-one-or-more-indexes-es-0-11/3426/10 "2017-07-06T04:18:11Z")