# I search same thing, but once can get and once can not get? **URL:** \ **Category:** Elasticsearch\ **Created:** [August 27, 2014, 8:35am UTC](https://discuss.elastic.co/t/i-search-same-thing-but-once-can-get-and-once-can-not-get/19486 "2014-08-27T08:35:50Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![fiefdx\_yang](https://avatars.discourse-cdn.com/v4/letter/f/bc8723/32.png) [@fiefdx\_yang](https://discuss.elastic.co/u/fiefdx_yang)\ **Post date:** [August 27, 2014, 8:35am UTC](https://discuss.elastic.co/t/i-search-same-thing-but-once-can-get-and-once-can-not-get/19486/1 "2014-08-27T08:35:50Z")
I use elasticsearch 1.2.1 with 1 cluster 16 nodes on one machine every node has data set refresh\_interval: 3600s _I am confused these:_ first I can get it [root@d04 scripts]# curl '[http://localhost:9200/avmlsearch/dynamic\_avml/\_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty](http://localhost:9200/avmlsearch/dynamic_avml/_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty)' { "took" : 2, "timed\_out" : false, "\_shards" : { "total" : 4, "successful" : 4, "failed" : 0 }, "hits" : { "total" : 1, "max\_score" : 1.6702784, "hits" : [ { "\_index" : "avmlsearch", "\_type" : "dynamic\_avml", "\_id" : "d48e4736f647c2f098972d71627dd09bc386c204", "\_score" : 1.6702784, "\_source":{"mutant": "", "file\_paths": "C:\WINDOWS\system32\usp10.dll\nC:\WINDOWS\system32\imm32.dll\nC:\WINDOWS\system32\lpk.dll\nC:\WINDOWS\system32\psapi.dll\nC:\WINDOWS\system32\ws2\_32.dll\nC:\WINDOWS\system32\ws2help.dll", "parent": "", "source": "dynamic\_avml", "genes": "", "windows\_api": "CreateToolhelp32Snapshot\nZwWriteFile\nCreateFileW\nZwTerminateProcess\nZwCreateFile\nGetModuleFileNameW\nReadFile\nLoadLibraryExW\nZwCreateSection", "ips": "", "urls": "", "crc32": "B8974180", "registries": "", "domains": "", "children": "", "import\_time": "2014-08-27 15:55:37.815 +0800", "md5": "263C7D7004704A3E321D2C6252D9C2A8"} } ] } } second I can not get it [root@d04 scripts]# curl '[http://localhost:9200/avmlsearch/dynamic\_avml/\_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty](http://localhost:9200/avmlsearch/dynamic_avml/_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty)' { "took" : 2, "timed\_out" : false, "\_shards" : { "total" : 4, "successful" : 4, "failed" : 0 }, "hits" : { "total" : 0, "max\_score" : null, "hits" : [] } } third I can get it [root@d04 scripts]# curl '[http://localhost:9200/avmlsearch/dynamic\_avml/\_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty](http://localhost:9200/avmlsearch/dynamic_avml/_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty)' { "took" : 2, "timed\_out" : false, "\_shards" : { "total" : 4, "successful" : 4, "failed" : 0 }, "hits" : { "total" : 1, "max\_score" : 1.6702784, "hits" : [ { "\_index" : "avmlsearch", "\_type" : "dynamic\_avml", "\_id" : "d48e4736f647c2f098972d71627dd09bc386c204", "\_score" : 1.6702784, "\_source":{"mutant": "", "file\_paths": "C:\WINDOWS\system32\usp10.dll\nC:\WINDOWS\system32\imm32.dll\nC:\WINDOWS\system32\lpk.dll\nC:\WINDOWS\system32\psapi.dll\nC:\WINDOWS\system32\ws2\_32.dll\nC:\WINDOWS\system32\ws2help.dll", "parent": "", "source": "dynamic\_avml", "genes": "", "windows\_api": "CreateToolhelp32Snapshot\nZwWriteFile\nCreateFileW\nZwTerminateProcess\nZwCreateFile\nGetModuleFileNameW\nReadFile\nLoadLibraryExW\nZwCreateSection", "ips": "", "urls": "", "crc32": "B8974180", "registries": "", "domains": "", "children": "", "import\_time": "2014-08-27 15:55:37.815 +0800", "md5": "263C7D7004704A3E321D2C6252D9C2A8"} } ] } } now, I can not get it too [root@d04 scripts]# curl '[http://localhost:9200/avmlsearch/dynamic\_avml/\_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty](http://localhost:9200/avmlsearch/dynamic_avml/_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty)' { "took" : 2, "timed\_out" : false, "\_shards" : { "total" : 4, "successful" : 4, "failed" : 0 }, "hits" : { "total" : 0, "max\_score" : null, "hits" : [] } } I do not know, how this happen??? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7a582c63-e163-446d-8fa9-91fa4d51d6d5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7a582c63-e163-446d-8fa9-91fa4d51d6d5%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Greg\_Murnane](https://avatars.discourse-cdn.com/v4/letter/g/b9bd4f/32.png) [@Greg\_Murnane](https://discuss.elastic.co/u/Greg_Murnane)\ **Post date:** [August 28, 2014, 10:41am UTC](https://discuss.elastic.co/t/i-search-same-thing-but-once-can-get-and-once-can-not-get/19486/2 "2014-08-28T10:41:22Z")
This is a symptom that could happen with bad GC events, or with split brain. Can you look at the GC logging output to see how long the stop the world pauses you're seeing are? You can also run a query like " curl -XGET '[http://localhost:9200/\_cluster/state/master\_node?local=true](http://localhost:9200/_cluster/state/master_node?local=true)' " on each of the nodes to make sure that they agree on which one is the master node. Look also at wait CPU and disk utilization when you run a query. Unless you have a physical disk for each node on this system, it's likely that there can be IO contention with 16 nodes querying the disks. If all that looks ok, if you are running replicas, then you can try pulling out a replica and an original, and loading them into an isolated ES node on another system, and query there. It's possible that some of the replicas could be corrupted, and this would allow you to detect that. * * * Out of curiousity, though, I wonder what the purpose of running so many nodes on a single machine is. ES is very effective at using the entire CPU with only one node, and replicating your heap size 16 times, adding IO contention, and splitting the cache 16 ways all seem like they would hurt performance immensely. -- The information transmitted in this email is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this email in error, please contact the sender and permanently delete the email from any computer. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f69c422b-7bbb-400b-a5c9-4f19eb0bd849%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f69c422b-7bbb-400b-a5c9-4f19eb0bd849%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [August 28, 2014, 4:13pm UTC](https://discuss.elastic.co/t/i-search-same-thing-but-once-can-get-and-once-can-not-get/19486/3 "2014-08-28T16:13:58Z")
Did you index with ES 1.2.0? Looks like the routing bug that was fixed in 1.2.1. If so, you must reindex your data. Jörg On Wed, Aug 27, 2014 at 10:35 AM, fiefdx yang [fiefdx@gmail.com](mailto:fiefdx@gmail.com) wrote: > I use elasticsearch 1.2.1 with > 1 cluster 16 nodes on one machine > every node has data > set refresh\_interval: 3600s > > _I am confused these:_ > first I can get it > [root@d04 scripts]# curl ' > [http://localhost:9200/avmlsearch/dynamic\_avml/\_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty](http://localhost:9200/avmlsearch/dynamic_avml/_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty) > ' > { > "took" : 2, > "timed\_out" : false, > "\_shards" : { > "total" : 4, > "successful" : 4, > "failed" : 0 > }, > "hits" : { > "total" : 1, > "max\_score" : 1.6702784, > "hits" : [ { > "\_index" : "avmlsearch", > "\_type" : "dynamic\_avml", > "\_id" : "d48e4736f647c2f098972d71627dd09bc386c204", > "\_score" : 1.6702784, > "\_source":{"mutant": "", "file\_paths": > "C:\WINDOWS\system32\usp10.dll\nC:\WINDOWS\system32\imm32.dll\nC:\WINDOWS\system32\lpk.dll\nC:\WINDOWS\system32\psapi.dll\nC:\WINDOWS\system32\ws2\_32.dll\nC:\WINDOWS\system32\ws2help.dll", > "parent": "", "source": "dynamic\_avml", "genes": "", "windows\_api": > "CreateToolhelp32Snapshot\nZwWriteFile\nCreateFileW\nZwTerminateProcess\nZwCreateFile\nGetModuleFileNameW\nReadFile\nLoadLibraryExW\nZwCreateSection", > "ips": "", "urls": "", "crc32": "B8974180", "registries": "", "domains": > "", "children": "", "import\_time": "2014-08-27 15:55:37.815 +0800", "md5": > "263C7D7004704A3E321D2C6252D9C2A8"} > } ] > } > } > second I can not get it > [root@d04 scripts]# curl ' > [http://localhost:9200/avmlsearch/dynamic\_avml/\_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty](http://localhost:9200/avmlsearch/dynamic_avml/_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty) > ' > { > "took" : 2, > "timed\_out" : false, > "\_shards" : { > "total" : 4, > "successful" : 4, > "failed" : 0 > }, > "hits" : { > "total" : 0, > "max\_score" : null, > "hits" : > } > } > third I can get it > [root@d04 scripts]# curl ' > [http://localhost:9200/avmlsearch/dynamic\_avml/\_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty](http://localhost:9200/avmlsearch/dynamic_avml/_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty) > ' > { > "took" : 2, > "timed\_out" : false, > "\_shards" : { > "total" : 4, > "successful" : 4, > "failed" : 0 > }, > "hits" : { > "total" : 1, > "max\_score" : 1.6702784, > "hits" : [ { > "\_index" : "avmlsearch", > "\_type" : "dynamic\_avml", > "\_id" : "d48e4736f647c2f098972d71627dd09bc386c204", > "\_score" : 1.6702784, > "\_source":{"mutant": "", "file\_paths": > "C:\WINDOWS\system32\usp10.dll\nC:\WINDOWS\system32\imm32.dll\nC:\WINDOWS\system32\lpk.dll\nC:\WINDOWS\system32\psapi.dll\nC:\WINDOWS\system32\ws2\_32.dll\nC:\WINDOWS\system32\ws2help.dll", > "parent": "", "source": "dynamic\_avml", "genes": "", "windows\_api": > "CreateToolhelp32Snapshot\nZwWriteFile\nCreateFileW\nZwTerminateProcess\nZwCreateFile\nGetModuleFileNameW\nReadFile\nLoadLibraryExW\nZwCreateSection", > "ips": "", "urls": "", "crc32": "B8974180", "registries": "", "domains": > "", "children": "", "import\_time": "2014-08-27 15:55:37.815 +0800", "md5": > "263C7D7004704A3E321D2C6252D9C2A8"} > } ] > } > } > now, I can not get it too > [root@d04 scripts]# curl ' > [http://localhost:9200/avmlsearch/dynamic\_avml/\_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty](http://localhost:9200/avmlsearch/dynamic_avml/_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty) > ' > { > "took" : 2, > "timed\_out" : false, > "\_shards" : { > "total" : 4, > "successful" : 4, > "failed" : 0 > }, > "hits" : { > "total" : 0, > "max\_score" : null, > "hits" : > } > } > > I do not know, how this happen??? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/7a582c63-e163-446d-8fa9-91fa4d51d6d5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7a582c63-e163-446d-8fa9-91fa4d51d6d5%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/7a582c63-e163-446d-8fa9-91fa4d51d6d5%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/7a582c63-e163-446d-8fa9-91fa4d51d6d5%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG-vodJSJ1mVKY%3DybsNc3hfz4QB9H3piA7EOFZEfcxGTw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG-vodJSJ1mVKY%3DybsNc3hfz4QB9H3piA7EOFZEfcxGTw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![fiefdx\_yang](https://avatars.discourse-cdn.com/v4/letter/f/bc8723/32.png) [@fiefdx\_yang](https://discuss.elastic.co/u/fiefdx_yang)\ **Post date:** [August 29, 2014, 8:45am UTC](https://discuss.elastic.co/t/i-search-same-thing-but-once-can-get-and-once-can-not-get/19486/4 "2014-08-29T08:45:11Z")
I can sure there is only one master node exists and 16 nodes work as one cluster. I think I know what it happened. I found that when successive requests, elasticsearch will execute query by once primary shards once replica shards at default configuration. But what I can not understand is why the primary shads and the replica shards given different result at the same time point? This happened when I index some new documents but not refresh, if I refresh the cluster, then the primary shards and the replica shards will give the same result. On Thursday, August 28, 2014 6:41:22 PM UTC+8, Greg Murnane wrote: > This is a symptom that could happen with bad GC events, or with split > brain. Can you look at the GC logging output to see how long the stop the > world pauses you're seeing are? You can also run a query like " curl -XGET ' > [http://localhost:9200/\_cluster/state/master\_node?local=true](http://localhost:9200/_cluster/state/master_node?local=true)' " on each of > the nodes to make sure that they agree on which one is the master node. > > Look also at wait CPU and disk utilization when you run a query. Unless > you have a physical disk for each node on this system, it's likely that > there can be IO contention with 16 nodes querying the disks. > > If all that looks ok, if you are running replicas, then you can try > pulling out a replica and an original, and loading them into an isolated ES > node on another system, and query there. It's possible that some of the > replicas could be corrupted, and this would allow you to detect that. > > * * * > > Out of curiousity, though, I wonder what the purpose of running so many > nodes on a single machine is. ES is very effective at using the entire CPU > with only one node, and replicating your heap size 16 times, adding IO > contention, and splitting the cache 16 ways all seem like they would hurt > performance immensely. > > The information transmitted in this email is intended only for the > person(s) or entity to which it is addressed and may contain confidential > and/or privileged material. Any review, retransmission, dissemination or > other use of, or taking of any action in reliance upon, this information by > persons or entities other than the intended recipient is prohibited. If you > received this email in error, please contact the sender and permanently > delete the email from any computer. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d86c4ac0-910f-4d00-8276-bec8aed220ad%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d86c4ac0-910f-4d00-8276-bec8aed220ad%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![fiefdx\_yang](https://avatars.discourse-cdn.com/v4/letter/f/bc8723/32.png) [@fiefdx\_yang](https://discuss.elastic.co/u/fiefdx_yang)\ **Post date:** [September 4, 2014, 2:44am UTC](https://discuss.elastic.co/t/i-search-same-thing-but-once-can-get-and-once-can-not-get/19486/5 "2014-09-04T02:44:41Z")
Finally，I find the answer. I set refresh\_interval: 3600s, but in the default, flush per 30min or trigger by translog size. Both refresh and flush can make new index doc can search. refresh and flush is timer by node, so every node have it own timer. In my cluster, one index use 8 nodes, set 4 shards and 1 replicas, so, 1 shard per node, so, primary and replica have different timer, so, primary and replica maybe have different result list when search the same query at the same time point. On Wednesday, August 27, 2014 4:35:50 PM UTC+8, fiefdx yang wrote: > I use elasticsearch 1.2.1 with > 1 cluster 16 nodes on one machine > every node has data > set refresh\_interval: 3600s > > _I am confused these:_ > first I can get it > [root@d04 scripts]# curl ' > [http://localhost:9200/avmlsearch/dynamic\_avml/\_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty](http://localhost:9200/avmlsearch/dynamic_avml/_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty) > ' > { > "took" : 2, > "timed\_out" : false, > "\_shards" : { > "total" : 4, > "successful" : 4, > "failed" : 0 > }, > "hits" : { > "total" : 1, > "max\_score" : 1.6702784, > "hits" : [ { > "\_index" : "avmlsearch", > "\_type" : "dynamic\_avml", > "\_id" : "d48e4736f647c2f098972d71627dd09bc386c204", > "\_score" : 1.6702784, > "\_source":{"mutant": "", "file\_paths": > "C:\WINDOWS\system32\usp10.dll\nC:\WINDOWS\system32\imm32.dll\nC:\WINDOWS\system32\lpk.dll\nC:\WINDOWS\system32\psapi.dll\nC:\WINDOWS\system32\ws2\_32.dll\nC:\WINDOWS\system32\ws2help.dll", > "parent": "", "source": "dynamic\_avml", "genes": "", "windows\_api": > "CreateToolhelp32Snapshot\nZwWriteFile\nCreateFileW\nZwTerminateProcess\nZwCreateFile\nGetModuleFileNameW\nReadFile\nLoadLibraryExW\nZwCreateSection", > "ips": "", "urls": "", "crc32": "B8974180", "registries": "", "domains": > "", "children": "", "import\_time": "2014-08-27 15:55:37.815 +0800", "md5": > "263C7D7004704A3E321D2C6252D9C2A8"} > } ] > } > } > second I can not get it > [root@d04 scripts]# curl ' > [http://localhost:9200/avmlsearch/dynamic\_avml/\_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty](http://localhost:9200/avmlsearch/dynamic_avml/_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty) > ' > { > "took" : 2, > "timed\_out" : false, > "\_shards" : { > "total" : 4, > "successful" : 4, > "failed" : 0 > }, > "hits" : { > "total" : 0, > "max\_score" : null, > "hits" : > } > } > third I can get it > [root@d04 scripts]# curl ' > [http://localhost:9200/avmlsearch/dynamic\_avml/\_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty](http://localhost:9200/avmlsearch/dynamic_avml/_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty) > ' > { > "took" : 2, > "timed\_out" : false, > "\_shards" : { > "total" : 4, > "successful" : 4, > "failed" : 0 > }, > "hits" : { > "total" : 1, > "max\_score" : 1.6702784, > "hits" : [ { > "\_index" : "avmlsearch", > "\_type" : "dynamic\_avml", > "\_id" : "d48e4736f647c2f098972d71627dd09bc386c204", > "\_score" : 1.6702784, > "\_source":{"mutant": "", "file\_paths": > "C:\WINDOWS\system32\usp10.dll\nC:\WINDOWS\system32\imm32.dll\nC:\WINDOWS\system32\lpk.dll\nC:\WINDOWS\system32\psapi.dll\nC:\WINDOWS\system32\ws2\_32.dll\nC:\WINDOWS\system32\ws2help.dll", > "parent": "", "source": "dynamic\_avml", "genes": "", "windows\_api": > "CreateToolhelp32Snapshot\nZwWriteFile\nCreateFileW\nZwTerminateProcess\nZwCreateFile\nGetModuleFileNameW\nReadFile\nLoadLibraryExW\nZwCreateSection", > "ips": "", "urls": "", "crc32": "B8974180", "registries": "", "domains": > "", "children": "", "import\_time": "2014-08-27 15:55:37.815 +0800", "md5": > "263C7D7004704A3E321D2C6252D9C2A8"} > } ] > } > } > now, I can not get it too > [root@d04 scripts]# curl ' > [http://localhost:9200/avmlsearch/dynamic\_avml/\_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty](http://localhost:9200/avmlsearch/dynamic_avml/_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty) > ' > { > "took" : 2, > "timed\_out" : false, > "\_shards" : { > "total" : 4, > "successful" : 4, > "failed" : 0 > }, > "hits" : { > "total" : 0, > "max\_score" : null, > "hits" : > } > } > > I do not know, how this happen??? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b92d569a-1a74-4232-ae61-d8df8683c0aa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b92d569a-1a74-4232-ae61-d8df8683c0aa%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![fiefdx\_yang](https://avatars.discourse-cdn.com/v4/letter/f/bc8723/32.png) [@fiefdx\_yang](https://discuss.elastic.co/u/fiefdx_yang)\ **Post date:** [September 4, 2014, 6:56am UTC](https://discuss.elastic.co/t/i-search-same-thing-but-once-can-get-and-once-can-not-get/19486/6 "2014-09-04T06:56:56Z")
Thanks, I think I find the answer, write at the bottom. On Thursday, August 28, 2014 6:41:22 PM UTC+8, Greg Murnane wrote: > This is a symptom that could happen with bad GC events, or with split > brain. Can you look at the GC logging output to see how long the stop the > world pauses you're seeing are? You can also run a query like " curl -XGET ' > [http://localhost:9200/\_cluster/state/master\_node?local=true](http://localhost:9200/_cluster/state/master_node?local=true)' " on each of > the nodes to make sure that they agree on which one is the master node. > > Look also at wait CPU and disk utilization when you run a query. Unless > you have a physical disk for each node on this system, it's likely that > there can be IO contention with 16 nodes querying the disks. > > If all that looks ok, if you are running replicas, then you can try > pulling out a replica and an original, and loading them into an isolated ES > node on another system, and query there. It's possible that some of the > replicas could be corrupted, and this would allow you to detect that. > > * * * > > Out of curiousity, though, I wonder what the purpose of running so many > nodes on a single machine is. ES is very effective at using the entire CPU > with only one node, and replicating your heap size 16 times, adding IO > contention, and splitting the cache 16 ways all seem like they would hurt > performance immensely. > > The information transmitted in this email is intended only for the > person(s) or entity to which it is addressed and may contain confidential > and/or privileged material. Any review, retransmission, dissemination or > other use of, or taking of any action in reliance upon, this information by > persons or entities other than the intended recipient is prohibited. If you > received this email in error, please contact the sender and permanently > delete the email from any computer. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1d3a72c6-6d94-4b62-9719-b1fbb5201eaf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1d3a72c6-6d94-4b62-9719-b1fbb5201eaf%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![fiefdx\_yang](https://avatars.discourse-cdn.com/v4/letter/f/bc8723/32.png) [@fiefdx\_yang](https://discuss.elastic.co/u/fiefdx_yang)\ **Post date:** [September 4, 2014, 6:57am UTC](https://discuss.elastic.co/t/i-search-same-thing-but-once-can-get-and-once-can-not-get/19486/7 "2014-09-04T06:57:18Z")
Thanks, I think I find the answer, write at the bottom. On Friday, August 29, 2014 12:14:11 AM UTC+8, Jörg Prante wrote: > Did you index with ES 1.2.0? Looks like the routing bug that was fixed in > 1.2.1. > > If so, you must reindex your data. > > Jörg > > On Wed, Aug 27, 2014 at 10:35 AM, fiefdx yang \<[fie...@gmail.com](mailto:fie...@gmail.com) > \\> wrote: > > > I use elasticsearch 1.2.1 with > > 1 cluster 16 nodes on one machine > > every node has data > > set refresh\_interval: 3600s > > > > _I am confused these:_ > > first I can get it > > [root@d04 scripts]# curl ' > > [http://localhost:9200/avmlsearch/dynamic\_avml/\_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty](http://localhost:9200/avmlsearch/dynamic_avml/_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty) > > ' > > { > > "took" : 2, > > "timed\_out" : false, > > "\_shards" : { > > "total" : 4, > > "successful" : 4, > > "failed" : 0 > > }, > > "hits" : { > > "total" : 1, > > "max\_score" : 1.6702784, > > "hits" : [ { > > "\_index" : "avmlsearch", > > "\_type" : "dynamic\_avml", > > "\_id" : "d48e4736f647c2f098972d71627dd09bc386c204", > > "\_score" : 1.6702784, > > "\_source":{"mutant": "", "file\_paths": > > "C:\WINDOWS\system32\usp10.dll\nC:\WINDOWS\system32\imm32.dll\nC:\WINDOWS\system32\lpk.dll\nC:\WINDOWS\system32\psapi.dll\nC:\WINDOWS\system32\ws2\_32.dll\nC:\WINDOWS\system32\ws2help.dll", > > "parent": "", "source": "dynamic\_avml", "genes": "", "windows\_api": > > "CreateToolhelp32Snapshot\nZwWriteFile\nCreateFileW\nZwTerminateProcess\nZwCreateFile\nGetModuleFileNameW\nReadFile\nLoadLibraryExW\nZwCreateSection", > > "ips": "", "urls": "", "crc32": "B8974180", "registries": "", "domains": > > "", "children": "", "import\_time": "2014-08-27 15:55:37.815 +0800", "md5": > > "263C7D7004704A3E321D2C6252D9C2A8"} > > } ] > > } > > } > > second I can not get it > > [root@d04 scripts]# curl ' > > [http://localhost:9200/avmlsearch/dynamic\_avml/\_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty](http://localhost:9200/avmlsearch/dynamic_avml/_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty) > > ' > > { > > "took" : 2, > > "timed\_out" : false, > > "\_shards" : { > > "total" : 4, > > "successful" : 4, > > "failed" : 0 > > }, > > "hits" : { > > "total" : 0, > > "max\_score" : null, > > "hits" : > > } > > } > > third I can get it > > [root@d04 scripts]# curl ' > > [http://localhost:9200/avmlsearch/dynamic\_avml/\_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty](http://localhost:9200/avmlsearch/dynamic_avml/_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty) > > ' > > { > > "took" : 2, > > "timed\_out" : false, > > "\_shards" : { > > "total" : 4, > > "successful" : 4, > > "failed" : 0 > > }, > > "hits" : { > > "total" : 1, > > "max\_score" : 1.6702784, > > "hits" : [ { > > "\_index" : "avmlsearch", > > "\_type" : "dynamic\_avml", > > "\_id" : "d48e4736f647c2f098972d71627dd09bc386c204", > > "\_score" : 1.6702784, > > "\_source":{"mutant": "", "file\_paths": > > "C:\WINDOWS\system32\usp10.dll\nC:\WINDOWS\system32\imm32.dll\nC:\WINDOWS\system32\lpk.dll\nC:\WINDOWS\system32\psapi.dll\nC:\WINDOWS\system32\ws2\_32.dll\nC:\WINDOWS\system32\ws2help.dll", > > "parent": "", "source": "dynamic\_avml", "genes": "", "windows\_api": > > "CreateToolhelp32Snapshot\nZwWriteFile\nCreateFileW\nZwTerminateProcess\nZwCreateFile\nGetModuleFileNameW\nReadFile\nLoadLibraryExW\nZwCreateSection", > > "ips": "", "urls": "", "crc32": "B8974180", "registries": "", "domains": > > "", "children": "", "import\_time": "2014-08-27 15:55:37.815 +0800", "md5": > > "263C7D7004704A3E321D2C6252D9C2A8"} > > } ] > > } > > } > > now, I can not get it too > > [root@d04 scripts]# curl ' > > [http://localhost:9200/avmlsearch/dynamic\_avml/\_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty](http://localhost:9200/avmlsearch/dynamic_avml/_search?q=263C7D7004704A3E321D2C6252D9C2A8&pretty) > > ' > > { > > "took" : 2, > > "timed\_out" : false, > > "\_shards" : { > > "total" : 4, > > "successful" : 4, > > "failed" : 0 > > }, > > "hits" : { > > "total" : 0, > > "max\_score" : null, > > "hits" : > > } > > } > > > > I do not know, how this happen??? > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/7a582c63-e163-446d-8fa9-91fa4d51d6d5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7a582c63-e163-446d-8fa9-91fa4d51d6d5%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/7a582c63-e163-446d-8fa9-91fa4d51d6d5%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/7a582c63-e163-446d-8fa9-91fa4d51d6d5%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4b411006-9bcd-42b8-bcfe-97d6776c1994%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4b411006-9bcd-42b8-bcfe-97d6776c1994%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:04am UTC](https://discuss.elastic.co/t/i-search-same-thing-but-once-can-get-and-once-can-not-get/19486/8 "2017-07-06T01:04:27Z")