# Two different results for the same id **URL:** \ **Category:** Elasticsearch\ **Created:** [March 9, 2014, 11:32am UTC](https://discuss.elastic.co/t/two-different-results-for-the-same-id/16253 "2014-03-09T11:32:34Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Tal\_Shemesh](https://avatars.discourse-cdn.com/v4/letter/t/7cd45c/32.png) [@Tal\_Shemesh](https://discuss.elastic.co/u/Tal_Shemesh)\ **Post date:** [March 9, 2014, 11:32am UTC](https://discuss.elastic.co/t/two-different-results-for-the-same-id/16253/1 "2014-03-09T11:32:34Z")
Hi, I have a cluster of 7 ES servers i am using it to hold the system servers config and for monitoring (with logstash and kibana) Three days ago, we started to see something strange. for the same request (id), we get different results. it looks like there are two objects in the system and the result is a round-robin between them i have checked and both answers has the same version 1. { "\_index": "config\_app", "\_type": "group", "\_id": "cl2s\_02", "\_version": 3, "exists": true, "\_source": { "feedMatcherConfig": { "maxPoolSize": 5000, "geo": { 1. { "\_index": "config\_app", "\_type": "group", "\_id": "cl2s\_02", "\_version": 3, "exists": true, "\_source": { "feedMatcherConfig": { "maxPoolSize": 5000, _"gidentifier": "cl2s\_02",_ "geo": { do you have any ideas what can i do? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [March 9, 2014, 11:50am UTC](https://discuss.elastic.co/t/two-different-results-for-the-same-id/16253/2 "2014-03-09T11:50:56Z")
What is the cluster state? I believe you are experiencing a split brain On Mar 9, 2014 11:32 AM, "Tal Shemesh" [tal.shemesh@gmail.com](mailto:tal.shemesh@gmail.com) wrote: > Hi, > I have a cluster of 7 ES servers > i am using it to hold the system servers config and for monitoring (with > logstash and kibana) > Three days ago, we started to see something strange. for the same request > (id), we get different results. > it looks like there are two objects in the system and the result is a > round-robin between them > i have checked and both answers has the same version > > 1. > > { > "\_index": "config\_app", > "\_type": "group", > "\_id": "cl2s\_02", > "\_version": 3, > "exists": true, > "\_source": { > "feedMatcherConfig": { > "maxPoolSize": 5000, > "geo": { > > 1. > > { > "\_index": "config\_app", > "\_type": "group", > "\_id": "cl2s\_02", > "\_version": 3, > "exists": true, > "\_source": { > "feedMatcherConfig": { > "maxPoolSize": 5000, > _"gidentifier": "cl2s\_02",_ > "geo": { > > do you have any ideas what can i do? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtbSJHHOCmA1HwgeZpqu%3DnB%3DTn6SMHJWJVpQKqU6W3U6g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtbSJHHOCmA1HwgeZpqu%3DnB%3DTn6SMHJWJVpQKqU6W3U6g%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Tal\_Shemesh](https://avatars.discourse-cdn.com/v4/letter/t/7cd45c/32.png) [@Tal\_Shemesh](https://discuss.elastic.co/u/Tal_Shemesh)\ **Post date:** [March 9, 2014, 1:18pm UTC](https://discuss.elastic.co/t/two-different-results-for-the-same-id/16253/3 "2014-03-09T13:18:31Z")
Hi Itamar, the cluster seams ok: cluster health: green (9, 12) also, i can see all the nodes and only one master. On Sunday, March 9, 2014 1:50:56 PM UTC+2, Itamar Syn-Hershko wrote: > What is the cluster state? I believe you are experiencing a split brain > On Mar 9, 2014 11:32 AM, "Tal Shemesh" \<[tal.s...@gmail.com](mailto:tal.s...@gmail.com) \\> > wrote: > > > Hi, > > I have a cluster of 7 ES servers > > i am using it to hold the system servers config and for monitoring (with > > logstash and kibana) > > Three days ago, we started to see something strange. for the same request > > (id), we get different results. > > it looks like there are two objects in the system and the result is a > > round-robin between them > > i have checked and both answers has the same version > > > > 1. > > > > { > > "\_index": "config\_app", > > "\_type": "group", > > "\_id": "cl2s\_02", > > "\_version": 3, > > "exists": true, > > "\_source": { > > "feedMatcherConfig": { > > "maxPoolSize": 5000, > > "geo": { > > > > 1. > > > > { > > "\_index": "config\_app", > > "\_type": "group", > > "\_id": "cl2s\_02", > > "\_version": 3, > > "exists": true, > > "\_source": { > > "feedMatcherConfig": { > > "maxPoolSize": 5000, > > _"gidentifier": "cl2s\_02",_ > > "geo": { > > > > do you have any ideas what can i do? > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5c08ad27-913c-4133-af7a-725237233a6b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5c08ad27-913c-4133-af7a-725237233a6b%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Daniel\_Gelbshtein](https://avatars.discourse-cdn.com/v4/letter/d/fbc32d/32.png) [@Daniel\_Gelbshtein](https://discuss.elastic.co/u/Daniel_Gelbshtein)\ **Post date:** [March 11, 2014, 11:25am UTC](https://discuss.elastic.co/t/two-different-results-for-the-same-id/16253/4 "2014-03-11T11:25:48Z")
Hi Itamar I work with Tal and we have a 2 data directories in the elasticsearch.yml configuration file. The exact line is : path.data: /var/lib/elasticsearch,/var/lib/elasticsearch2 Some of the servers in the cluster contain sub folders in the /var/lib/elasticsearch2 directory and some don't. The ones that don't I'm guessing, never contained a shard in the folder. Could this contribute to the problem we're having? On Sunday, 9 March 2014 15:18:31 UTC+2, Tal Shemesh wrote: > Hi Itamar, > > the cluster seams ok: > > cluster health: green (9, 12) > > also, i can see all the nodes and only one master. > > On Sunday, March 9, 2014 1:50:56 PM UTC+2, Itamar Syn-Hershko wrote: > > > What is the cluster state? I believe you are experiencing a split brain > > On Mar 9, 2014 11:32 AM, "Tal Shemesh" [tal.s...@gmail.com](mailto:tal.s...@gmail.com) wrote: > > > > > Hi, > > > I have a cluster of 7 ES servers > > > i am using it to hold the system servers config and for monitoring (with > > > logstash and kibana) > > > Three days ago, we started to see something strange. for the same > > > request (id), we get different results. > > > it looks like there are two objects in the system and the result is a > > > round-robin between them > > > i have checked and both answers has the same version > > > > > > 1. > > > > > > { > > > "\_index": "config\_app", > > > "\_type": "group", > > > "\_id": "cl2s\_02", > > > "\_version": 3, > > > "exists": true, > > > "\_source": { > > > "feedMatcherConfig": { > > > "maxPoolSize": 5000, > > > "geo": { > > > > > > 1. > > > > > > { > > > "\_index": "config\_app", > > > "\_type": "group", > > > "\_id": "cl2s\_02", > > > "\_version": 3, > > > "exists": true, > > > "\_source": { > > > "feedMatcherConfig": { > > > "maxPoolSize": 5000, > > > _"gidentifier": "cl2s\_02",_ > > > "geo": { > > > > > > do you have any ideas what can i do? > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6fbc82bb-7ba3-419b-b473-aa8ea6f83406%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6fbc82bb-7ba3-419b-b473-aa8ea6f83406%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Boaz\_Leskes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boaz_leskes/32/723_2.png) [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)\ **Post date:** [March 11, 2014, 11:55am UTC](https://discuss.elastic.co/t/two-different-results-for-the-same-id/16253/5 "2014-03-11T11:55:48Z")
Daniel, Do you use custom routing? Do you change to async replication? Cheers, Boaz On Tuesday, March 11, 2014 12:25:48 PM UTC+1, Daniel Gelbshtein wrote: > Hi Itamar > > I work with Tal and we have a 2 data directories in the elasticsearch.yml > configuration file. > The exact line is : > path.data: /var/lib/elasticsearch,/var/lib/elasticsearch2 > > Some of the servers in the cluster contain sub folders in the > /var/lib/elasticsearch2 directory and some don't. > The ones that don't I'm guessing, never contained a shard in the folder. > > Could this contribute to the problem we're having? > > On Sunday, 9 March 2014 15:18:31 UTC+2, Tal Shemesh wrote: > > > Hi Itamar, > > > > the cluster seams ok: > > > > cluster health: green (9, 12) > > > > also, i can see all the nodes and only one master. > > > > On Sunday, March 9, 2014 1:50:56 PM UTC+2, Itamar Syn-Hershko wrote: > > > > > What is the cluster state? I believe you are experiencing a split brain > > > On Mar 9, 2014 11:32 AM, "Tal Shemesh" [tal.s...@gmail.com](mailto:tal.s...@gmail.com) wrote: > > > > > > > Hi, > > > > I have a cluster of 7 ES servers > > > > i am using it to hold the system servers config and for monitoring > > > > (with logstash and kibana) > > > > Three days ago, we started to see something strange. for the same > > > > request (id), we get different results. > > > > it looks like there are two objects in the system and the result is a > > > > round-robin between them > > > > i have checked and both answers has the same version > > > > > > > > 1. > > > > > > > > { > > > > "\_index": "config\_app", > > > > "\_type": "group", > > > > "\_id": "cl2s\_02", > > > > "\_version": 3, > > > > "exists": true, > > > > "\_source": { > > > > "feedMatcherConfig": { > > > > "maxPoolSize": 5000, > > > > "geo": { > > > > > > > > 1. > > > > > > > > { > > > > "\_index": "config\_app", > > > > "\_type": "group", > > > > "\_id": "cl2s\_02", > > > > "\_version": 3, > > > > "exists": true, > > > > "\_source": { > > > > "feedMatcherConfig": { > > > > "maxPoolSize": 5000, > > > > _"gidentifier": "cl2s\_02",_ > > > > "geo": { > > > > > > > > do you have any ideas what can i do? > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/28a59e91-f61b-4718-97bc-caeb371e4e42%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/28a59e91-f61b-4718-97bc-caeb371e4e42%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Daniel\_Gelbshtein](https://avatars.discourse-cdn.com/v4/letter/d/fbc32d/32.png) [@Daniel\_Gelbshtein](https://discuss.elastic.co/u/Daniel_Gelbshtein)\ **Post date:** [March 11, 2014, 12:23pm UTC](https://discuss.elastic.co/t/two-different-results-for-the-same-id/16253/6 "2014-03-11T12:23:39Z")
Hi Boaz, No custom routing. We use the default replication method implemented in version 0.90.11 The \*"gidentifier": "cl2s\_02" \*from Tals original post, is a couple of days old. So it should have already replicated to all shards in the cluster. On Tuesday, 11 March 2014 13:55:48 UTC+2, Boaz Leskes wrote: > Daniel, > > Do you use custom routing? Do you change to async replication? > > Cheers, > Boaz > > On Tuesday, March 11, 2014 12:25:48 PM UTC+1, Daniel Gelbshtein wrote: > > > Hi Itamar > > > > I work with Tal and we have a 2 data directories in the elasticsearch.yml > > configuration file. > > The exact line is : > > path.data: /var/lib/elasticsearch,/var/lib/elasticsearch2 > > > > Some of the servers in the cluster contain sub folders in the > > /var/lib/elasticsearch2 directory and some don't. > > The ones that don't I'm guessing, never contained a shard in the folder. > > > > Could this contribute to the problem we're having? > > > > On Sunday, 9 March 2014 15:18:31 UTC+2, Tal Shemesh wrote: > > > > > Hi Itamar, > > > > > > the cluster seams ok: > > > > > > cluster health: green (9, 12) > > > > > > also, i can see all the nodes and only one master. > > > > > > On Sunday, March 9, 2014 1:50:56 PM UTC+2, Itamar Syn-Hershko wrote: > > > > > > > What is the cluster state? I believe you are experiencing a split brain > > > > On Mar 9, 2014 11:32 AM, "Tal Shemesh" [tal.s...@gmail.com](mailto:tal.s...@gmail.com) wrote: > > > > > > > > > Hi, > > > > > I have a cluster of 7 ES servers > > > > > i am using it to hold the system servers config and for monitoring > > > > > (with logstash and kibana) > > > > > Three days ago, we started to see something strange. for the same > > > > > request (id), we get different results. > > > > > it looks like there are two objects in the system and the result is a > > > > > round-robin between them > > > > > i have checked and both answers has the same version > > > > > > > > > > 1. > > > > > > > > > > { > > > > > "\_index": "config\_app", > > > > > "\_type": "group", > > > > > "\_id": "cl2s\_02", > > > > > "\_version": 3, > > > > > "exists": true, > > > > > "\_source": { > > > > > "feedMatcherConfig": { > > > > > "maxPoolSize": 5000, > > > > > "geo": { > > > > > > > > > > 1. > > > > > > > > > > { > > > > > "\_index": "config\_app", > > > > > "\_type": "group", > > > > > "\_id": "cl2s\_02", > > > > > "\_version": 3, > > > > > "exists": true, > > > > > "\_source": { > > > > > "feedMatcherConfig": { > > > > > "maxPoolSize": 5000, > > > > > _"gidentifier": "cl2s\_02",_ > > > > > "geo": { > > > > > > > > > > do you have any ideas what can i do? > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/59a13dc6-7c7a-4891-a170-eb458baa6427%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/59a13dc6-7c7a-4891-a170-eb458baa6427%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [March 11, 2014, 12:51pm UTC](https://discuss.elastic.co/t/two-different-results-for-the-same-id/16253/7 "2014-03-11T12:51:08Z")
I believe this to be the culprit, yes (and thus - a bug in Elasticsearch). The question of subfolders you may or may not have under each is simply a question of how elasticsearch decided to split the data between the folders. Is the only difference you are seeing between docs is this one field existing/missing? I'd try and characterize this and perhaps figure out a way to create minimal repro the ES team can run. Also, I'd hope there's a way to merge multiple data folders into one so you could try that and see if it indeed the cause for this problem Cheers, -- Itamar Syn-Hershko [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) Freelance Developer & Consultant Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) On Tue, Mar 11, 2014 at 11:25 AM, Daniel Gelbshtein \< [daniel.kantorov@gmail.com](mailto:daniel.kantorov@gmail.com)\> wrote: > Hi Itamar > > I work with Tal and we have a 2 data directories in the elasticsearch.yml > configuration file. > The exact line is : > path.data: /var/lib/elasticsearch,/var/lib/elasticsearch2 > > Some of the servers in the cluster contain sub folders in the > /var/lib/elasticsearch2 directory and some don't. > The ones that don't I'm guessing, never contained a shard in the folder. > > Could this contribute to the problem we're having? > > On Sunday, 9 March 2014 15:18:31 UTC+2, Tal Shemesh wrote: > > > Hi Itamar, > > > > the cluster seams ok: > > > > cluster health: green (9, 12) > > > > also, i can see all the nodes and only one master. > > > > On Sunday, March 9, 2014 1:50:56 PM UTC+2, Itamar Syn-Hershko wrote: > > > > > What is the cluster state? I believe you are experiencing a split brain > > > On Mar 9, 2014 11:32 AM, "Tal Shemesh" [tal.s...@gmail.com](mailto:tal.s...@gmail.com) wrote: > > > > > > > Hi, > > > > I have a cluster of 7 ES servers > > > > i am using it to hold the system servers config and for monitoring > > > > (with logstash and kibana) > > > > Three days ago, we started to see something strange. for the same > > > > request (id), we get different results. > > > > it looks like there are two objects in the system and the result is a > > > > round-robin between them > > > > i have checked and both answers has the same version > > > > > > > > 1. > > > > > > > > { > > > > "\_index": "config\_app", > > > > "\_type": "group", > > > > "\_id": "cl2s\_02", > > > > "\_version": 3, > > > > "exists": true, > > > > "\_source": { > > > > "feedMatcherConfig": { > > > > "maxPoolSize": 5000, > > > > "geo": { > > > > > > > > 1. > > > > > > > > { > > > > "\_index": "config\_app", > > > > "\_type": "group", > > > > "\_id": "cl2s\_02", > > > > "\_version": 3, > > > > "exists": true, > > > > "\_source": { > > > > "feedMatcherConfig": { > > > > "maxPoolSize": 5000, > > > > _"gidentifier": "cl2s\_02",_ > > > > "geo": { > > > > > > > > do you have any ideas what can i do? > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9% > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/6fbc82bb-7ba3-419b-b473-aa8ea6f83406%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6fbc82bb-7ba3-419b-b473-aa8ea6f83406%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/6fbc82bb-7ba3-419b-b473-aa8ea6f83406%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6fbc82bb-7ba3-419b-b473-aa8ea6f83406%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zu%2BBdwEXEaUgWz%3DmLLXrc3X79DLud0%2BThnChBz07K7kww%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zu%2BBdwEXEaUgWz%3DmLLXrc3X79DLud0%2BThnChBz07K7kww%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Daniel\_Gelbshtein](https://avatars.discourse-cdn.com/v4/letter/d/fbc32d/32.png) [@Daniel\_Gelbshtein](https://discuss.elastic.co/u/Daniel_Gelbshtein)\ **Post date:** [March 11, 2014, 2:04pm UTC](https://discuss.elastic.co/t/two-different-results-for-the-same-id/16253/8 "2014-03-11T14:04:40Z")
In this particular doc, yes. This is the only difference. But there are other docs that have a 2 or fields missing/changed. On Tuesday, 11 March 2014 14:51:08 UTC+2, Itamar Syn-Hershko wrote: > I believe this to be the culprit, yes (and thus - a bug in Elasticsearch). > > The question of subfolders you may or may not have under each is simply a > question of how elasticsearch decided to split the data between the folders. > > Is the only difference you are seeing between docs is this one field > existing/missing? I'd try and characterize this and perhaps figure out a > way to create minimal repro the ES team can run. Also, I'd hope there's a > way to merge multiple data folders into one so you could try that and see > if it indeed the cause for this problem > > Cheers, > > -- > > Itamar Syn-Hershko > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > Freelance Developer & Consultant > Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) > > On Tue, Mar 11, 2014 at 11:25 AM, Daniel Gelbshtein \<[daniel....@gmail.com](mailto:daniel....@gmail.com)\ > > > wrote: > > > Hi Itamar > > > > I work with Tal and we have a 2 data directories in the elasticsearch.yml > > configuration file. > > The exact line is : > > path.data: /var/lib/elasticsearch,/var/lib/elasticsearch2 > > > > Some of the servers in the cluster contain sub folders in the > > /var/lib/elasticsearch2 directory and some don't. > > The ones that don't I'm guessing, never contained a shard in the folder. > > > > Could this contribute to the problem we're having? > > > > On Sunday, 9 March 2014 15:18:31 UTC+2, Tal Shemesh wrote: > > > > > Hi Itamar, > > > > > > the cluster seams ok: > > > > > > cluster health: green (9, 12) > > > > > > also, i can see all the nodes and only one master. > > > > > > On Sunday, March 9, 2014 1:50:56 PM UTC+2, Itamar Syn-Hershko wrote: > > > > > > > What is the cluster state? I believe you are experiencing a split brain > > > > On Mar 9, 2014 11:32 AM, "Tal Shemesh" [tal.s...@gmail.com](mailto:tal.s...@gmail.com) wrote: > > > > > > > > > Hi, > > > > > I have a cluster of 7 ES servers > > > > > i am using it to hold the system servers config and for monitoring > > > > > (with logstash and kibana) > > > > > Three days ago, we started to see something strange. for the same > > > > > request (id), we get different results. > > > > > it looks like there are two objects in the system and the result is a > > > > > round-robin between them > > > > > i have checked and both answers has the same version > > > > > > > > > > 1. > > > > > > > > > > { > > > > > "\_index": "config\_app", > > > > > "\_type": "group", > > > > > "\_id": "cl2s\_02", > > > > > "\_version": 3, > > > > > "exists": true, > > > > > "\_source": { > > > > > "feedMatcherConfig": { > > > > > "maxPoolSize": 5000, > > > > > "geo": { > > > > > > > > > > 1. > > > > > > > > > > { > > > > > "\_index": "config\_app", > > > > > "\_type": "group", > > > > > "\_id": "cl2s\_02", > > > > > "\_version": 3, > > > > > "exists": true, > > > > > "\_source": { > > > > > "feedMatcherConfig": { > > > > > "maxPoolSize": 5000, > > > > > _"gidentifier": "cl2s\_02",_ > > > > > "geo": { > > > > > > > > > > do you have any ideas what can i do? > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9% > > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google Groups > > > > "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send an > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/6fbc82bb-7ba3-419b-b473-aa8ea6f83406%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6fbc82bb-7ba3-419b-b473-aa8ea6f83406%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/6fbc82bb-7ba3-419b-b473-aa8ea6f83406%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6fbc82bb-7ba3-419b-b473-aa8ea6f83406%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1224de2d-6fb4-4570-abe4-3546eaf0f014%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1224de2d-6fb4-4570-abe4-3546eaf0f014%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Daniel\_Gelbshtein](https://avatars.discourse-cdn.com/v4/letter/d/fbc32d/32.png) [@Daniel\_Gelbshtein](https://discuss.elastic.co/u/Daniel_Gelbshtein)\ **Post date:** [March 11, 2014, 2:06pm UTC](https://discuss.elastic.co/t/two-different-results-for-the-same-id/16253/9 "2014-03-11T14:06:13Z")
Thank you for the replies. In this particular doc, yes this is the only difference. But there are a few more that have at least 2 fields that are missing/changed. On Tuesday, 11 March 2014 14:51:08 UTC+2, Itamar Syn-Hershko wrote: > I believe this to be the culprit, yes (and thus - a bug in Elasticsearch). > > The question of subfolders you may or may not have under each is simply a > question of how elasticsearch decided to split the data between the folders. > > Is the only difference you are seeing between docs is this one field > existing/missing? I'd try and characterize this and perhaps figure out a > way to create minimal repro the ES team can run. Also, I'd hope there's a > way to merge multiple data folders into one so you could try that and see > if it indeed the cause for this problem > > Cheers, > > -- > > Itamar Syn-Hershko > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) > Freelance Developer & Consultant > Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) > > On Tue, Mar 11, 2014 at 11:25 AM, Daniel Gelbshtein \<[daniel....@gmail.com](mailto:daniel....@gmail.com)\ > > > wrote: > > > Hi Itamar > > > > I work with Tal and we have a 2 data directories in the elasticsearch.yml > > configuration file. > > The exact line is : > > path.data: /var/lib/elasticsearch,/var/lib/elasticsearch2 > > > > Some of the servers in the cluster contain sub folders in the > > /var/lib/elasticsearch2 directory and some don't. > > The ones that don't I'm guessing, never contained a shard in the folder. > > > > Could this contribute to the problem we're having? > > > > On Sunday, 9 March 2014 15:18:31 UTC+2, Tal Shemesh wrote: > > > > > Hi Itamar, > > > > > > the cluster seams ok: > > > > > > cluster health: green (9, 12) > > > > > > also, i can see all the nodes and only one master. > > > > > > On Sunday, March 9, 2014 1:50:56 PM UTC+2, Itamar Syn-Hershko wrote: > > > > > > > What is the cluster state? I believe you are experiencing a split brain > > > > On Mar 9, 2014 11:32 AM, "Tal Shemesh" [tal.s...@gmail.com](mailto:tal.s...@gmail.com) wrote: > > > > > > > > > Hi, > > > > > I have a cluster of 7 ES servers > > > > > i am using it to hold the system servers config and for monitoring > > > > > (with logstash and kibana) > > > > > Three days ago, we started to see something strange. for the same > > > > > request (id), we get different results. > > > > > it looks like there are two objects in the system and the result is a > > > > > round-robin between them > > > > > i have checked and both answers has the same version > > > > > > > > > > 1. > > > > > > > > > > { > > > > > "\_index": "config\_app", > > > > > "\_type": "group", > > > > > "\_id": "cl2s\_02", > > > > > "\_version": 3, > > > > > "exists": true, > > > > > "\_source": { > > > > > "feedMatcherConfig": { > > > > > "maxPoolSize": 5000, > > > > > "geo": { > > > > > > > > > > 1. > > > > > > > > > > { > > > > > "\_index": "config\_app", > > > > > "\_type": "group", > > > > > "\_id": "cl2s\_02", > > > > > "\_version": 3, > > > > > "exists": true, > > > > > "\_source": { > > > > > "feedMatcherConfig": { > > > > > "maxPoolSize": 5000, > > > > > _"gidentifier": "cl2s\_02",_ > > > > > "geo": { > > > > > > > > > > do you have any ideas what can i do? > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9% > > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0f7cf7f7-bcb9-456f-b2c5-57fd2ae9dbd9%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google Groups > > > > "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send an > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/6fbc82bb-7ba3-419b-b473-aa8ea6f83406%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6fbc82bb-7ba3-419b-b473-aa8ea6f83406%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/6fbc82bb-7ba3-419b-b473-aa8ea6f83406%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6fbc82bb-7ba3-419b-b473-aa8ea6f83406%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/fe915715-9eef-4d60-af51-4c35a48291b7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fe915715-9eef-4d60-af51-4c35a48291b7%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:44am UTC](https://discuss.elastic.co/t/two-different-results-for-the-same-id/16253/10 "2017-07-06T01:44:10Z")