# Search Periodically Failing ... Even After Many Refresh Intervals **URL:** \ **Category:** Elasticsearch\ **Created:** [March 1, 2013, 3:24am UTC](https://discuss.elastic.co/t/search-periodically-failing-even-after-many-refresh-intervals/10962 "2013-03-01T03:24:11Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![amos\_wood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amos_wood/32/1880_2.png) [@amos\_wood](https://discuss.elastic.co/u/amos_wood)\ **Post date:** [March 1, 2013, 3:24am UTC](https://discuss.elastic.co/t/search-periodically-failing-even-after-many-refresh-intervals/10962/1 "2013-03-01T03:24:11Z")
Here is my search use case which seems to be failing periodically: 1. Create an index with a 500ms refresh interval 2. Write a document to an index 3. Pause for between 500ms and 5s 4. Attempt to search for a value in 1 field 5. Attempt to search again for the same value in the same field In my attempt to search, sometimes I am getting back no results on the first try with the second search attempt succeeding. Other times, I am getting back results on both searches. Even more strange, sometimes I get back a result on the first search but not on the second search. Does anyone know what may be going on? I have a gist script which will simulate the issue. I could run this gist on our production environment cluster and see the issue happen about every other run. When I run it on our sandbox environment, I cannot reproduce the issue. > _Production Cluster Configuration:_ ElasticSearch Version 0.19.4 1 Master Non Data Node 1 Master Data Node 6 Data Nodes All VM servers with 4 CPUs with 18GB memory. _Sandbox Cluster Configuration:_ ElasticSearch Version 0.19.4 2 Master Data Nodes Both VM servers with 2 CPUs with 8GB memory. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![simonw\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonw_2/32/1130_2.png) [@simonw\_2](https://discuss.elastic.co/u/simonw_2)\ **Post date:** [March 1, 2013, 9:22am UTC](https://discuss.elastic.co/t/search-periodically-failing-even-after-many-refresh-intervals/10962/2 "2013-03-01T09:22:05Z")
On Friday, March 1, 2013 4:24:11 AM UTC+1, amos.wood wrote: > Here is my search use case which seems to be failing periodically: > > 1. Create an index with a 500ms refresh interval > 2. Write a document to an index > 3. Pause for between 500ms and 5s > 4. Attempt to search for a value in 1 field > 5. Attempt to search again for the same value in the same field > > In my attempt to search, sometimes I am getting back no results on the > first try with the second search attempt succeeding. Other times, I am > getting back results on both searches. Even more strange, sometimes I get > back a result on the first search but not on the second search. Does > anyone know what may be going on? Hey, the refresh\_interval is not guaranteeing to make results visible 500ms after you write the document to the index. it depends on many things like how long it takes the do the refresh etc. so the interval can be higher in practice. If you want to run this testcase you should make sure to call refresh explicitly via the rest API to make sure that everything has been refreshed. simon > I have a gist script which will simulate the issue. I could run this gist > on our production environment cluster and see the issue happen about every > other run. When I run it on our sandbox environment, I cannot reproduce > the issue. > > [ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub](https://gist.github.com/anonymous/78e12092e648e96c20f0) > > _Production Cluster Configuration:_ > > Elasticsearch Version 0.19.4 > > 1 Master Non Data Node > 1 Master Data Node > 6 Data Nodes > > All VM servers with 4 CPUs with 18GB memory. > > _Sandbox Cluster Configuration:_ > Elasticsearch Version 0.19.4 > > 2 Master Data Nodes > > Both VM servers with 2 CPUs with 8GB memory. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![amos\_wood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amos_wood/32/1880_2.png) [@amos\_wood](https://discuss.elastic.co/u/amos_wood)\ **Post date:** [March 1, 2013, 11:46am UTC](https://discuss.elastic.co/t/search-periodically-failing-even-after-many-refresh-intervals/10962/3 "2013-03-01T11:46:22Z")
Simon, It makes sense to explicitly call the refresh. If you look at my gist, I am doing that with the 3rd record. When I explictly call the refresh, the search always return. It also makes sense that a refresh that is triggered intravenously through the use of the refresh\_interval could actually take longer than the 500ms that it is triggered with. However in my test, I have the refresh interval set to 500ms and I am pausing for 5 seconds and the search is not coming back. I have actually seen the same thing happen when pausing for 10 seconds. Then after waiting for 5 or 10 seconds, searching, and then immediately searching again and then getting the results is not right. I could see the search not returning in 600ms, but 10s later on a new index which I just created or with no other things going on with the cluster, it seems like something is wrong. Note: I am only seeing the problem on a cluster which can support a replicas count of 2 for the index. It seems like in the round-robin search approach, that the first /shard/replica that it is searching hasn't been indexed while the second searched shard/replica does have it indexed. After 10 seconds though, I don't know why everything wouldn't be indexed on the cluster though. Amos On Friday, March 1, 2013 3:22:05 AM UTC-6, simonw wrote: > On Friday, March 1, 2013 4:24:11 AM UTC+1, amos.wood wrote: > > > Here is my search use case which seems to be failing periodically: > > > > 1. Create an index with a 500ms refresh interval > > 2. Write a document to an index > > 3. Pause for between 500ms and 5s > > 4. Attempt to search for a value in 1 field > > 5. Attempt to search again for the same value in the same field > > > > In my attempt to search, sometimes I am getting back no results on the > > first try with the second search attempt succeeding. Other times, I am > > getting back results on both searches. Even more strange, sometimes I get > > back a result on the first search but not on the second search. Does > > anyone know what may be going on? > > Hey, > > the refresh\_interval is not guaranteeing to make results visible 500ms > after you write the document to the index. it depends on many things like > how long it takes the do the refresh etc. so the interval can be higher in > practice. If you want to run this testcase you should make sure to call > refresh explicitly via the rest API to make sure that everything has been > refreshed. > > simon > > > I have a gist script which will simulate the issue. I could run this > > gist on our production environment cluster and see the issue happen about > > every other run. When I run it on our sandbox environment, I cannot > > reproduce the issue. > > > > [ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub](https://gist.github.com/anonymous/78e12092e648e96c20f0) > > > > _Production Cluster Configuration:_ > > > > Elasticsearch Version 0.19.4 > > > > 1 Master Non Data Node > > 1 Master Data Node > > 6 Data Nodes > > > > All VM servers with 4 CPUs with 18GB memory. > > > > _Sandbox Cluster Configuration:_ > > Elasticsearch Version 0.19.4 > > > > 2 Master Data Nodes > > > > Both VM servers with 2 CPUs with 8GB memory. On Friday, March 1, 2013 3:22:05 AM UTC-6, simonw wrote: > On Friday, March 1, 2013 4:24:11 AM UTC+1, amos.wood wrote: > > > Here is my search use case which seems to be failing periodically: > > > > 1. Create an index with a 500ms refresh interval > > 2. Write a document to an index > > 3. Pause for between 500ms and 5s > > 4. Attempt to search for a value in 1 field > > 5. Attempt to search again for the same value in the same field > > > > In my attempt to search, sometimes I am getting back no results on the > > first try with the second search attempt succeeding. Other times, I am > > getting back results on both searches. Even more strange, sometimes I get > > back a result on the first search but not on the second search. Does > > anyone know what may be going on? > > Hey, > > the refresh\_interval is not guaranteeing to make results visible 500ms > after you write the document to the index. it depends on many things like > how long it takes the do the refresh etc. so the interval can be higher in > practice. If you want to run this testcase you should make sure to call > refresh explicitly via the rest API to make sure that everything has been > refreshed. > > simon > > > I have a gist script which will simulate the issue. I could run this > > gist on our production environment cluster and see the issue happen about > > every other run. When I run it on our sandbox environment, I cannot > > reproduce the issue. > > > > [ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub](https://gist.github.com/anonymous/78e12092e648e96c20f0) > > > > _Production Cluster Configuration:_ > > > > Elasticsearch Version 0.19.4 > > > > 1 Master Non Data Node > > 1 Master Data Node > > 6 Data Nodes > > > > All VM servers with 4 CPUs with 18GB memory. > > > > _Sandbox Cluster Configuration:_ > > Elasticsearch Version 0.19.4 > > > > 2 Master Data Nodes > > > > Both VM servers with 2 CPUs with 8GB memory. On Friday, March 1, 2013 3:22:05 AM UTC-6, simonw wrote: > On Friday, March 1, 2013 4:24:11 AM UTC+1, amos.wood wrote: > > > Here is my search use case which seems to be failing periodically: > > > > 1. Create an index with a 500ms refresh interval > > 2. Write a document to an index > > 3. Pause for between 500ms and 5s > > 4. Attempt to search for a value in 1 field > > 5. Attempt to search again for the same value in the same field > > > > In my attempt to search, sometimes I am getting back no results on the > > first try with the second search attempt succeeding. Other times, I am > > getting back results on both searches. Even more strange, sometimes I get > > back a result on the first search but not on the second search. Does > > anyone know what may be going on? > > Hey, > > the refresh\_interval is not guaranteeing to make results visible 500ms > after you write the document to the index. it depends on many things like > how long it takes the do the refresh etc. so the interval can be higher in > practice. If you want to run this testcase you should make sure to call > refresh explicitly via the rest API to make sure that everything has been > refreshed. > > simon > > > I have a gist script which will simulate the issue. I could run this > > gist on our production environment cluster and see the issue happen about > > every other run. When I run it on our sandbox environment, I cannot > > reproduce the issue. > > > > [ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub](https://gist.github.com/anonymous/78e12092e648e96c20f0) > > > > _Production Cluster Configuration:_ > > > > Elasticsearch Version 0.19.4 > > > > 1 Master Non Data Node > > 1 Master Data Node > > 6 Data Nodes > > > > All VM servers with 4 CPUs with 18GB memory. > > > > _Sandbox Cluster Configuration:_ > > Elasticsearch Version 0.19.4 > > > > 2 Master Data Nodes > > > > Both VM servers with 2 CPUs with 8GB memory. On Friday, March 1, 2013 3:22:05 AM UTC-6, simonw wrote: > On Friday, March 1, 2013 4:24:11 AM UTC+1, amos.wood wrote: > > > Here is my search use case which seems to be failing periodically: > > > > 1. Create an index with a 500ms refresh interval > > 2. Write a document to an index > > 3. Pause for between 500ms and 5s > > 4. Attempt to search for a value in 1 field > > 5. Attempt to search again for the same value in the same field > > > > In my attempt to search, sometimes I am getting back no results on the > > first try with the second search attempt succeeding. Other times, I am > > getting back results on both searches. Even more strange, sometimes I get > > back a result on the first search but not on the second search. Does > > anyone know what may be going on? > > Hey, > > the refresh\_interval is not guaranteeing to make results visible 500ms > after you write the document to the index. it depends on many things like > how long it takes the do the refresh etc. so the interval can be higher in > practice. If you want to run this testcase you should make sure to call > refresh explicitly via the rest API to make sure that everything has been > refreshed. > > simon > > > I have a gist script which will simulate the issue. I could run this > > gist on our production environment cluster and see the issue happen about > > every other run. When I run it on our sandbox environment, I cannot > > reproduce the issue. > > > > [ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub](https://gist.github.com/anonymous/78e12092e648e96c20f0) > > > > _Production Cluster Configuration:_ > > > > Elasticsearch Version 0.19.4 > > > > 1 Master Non Data Node > > 1 Master Data Node > > 6 Data Nodes > > > > All VM servers with 4 CPUs with 18GB memory. > > > > _Sandbox Cluster Configuration:_ > > Elasticsearch Version 0.19.4 > > > > 2 Master Data Nodes > > > > Both VM servers with 2 CPUs with 8GB memory. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![simonw\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonw_2/32/1130_2.png) [@simonw\_2](https://discuss.elastic.co/u/simonw_2)\ **Post date:** [March 1, 2013, 12:17pm UTC](https://discuss.elastic.co/t/search-periodically-failing-even-after-many-refresh-intervals/10962/4 "2013-03-01T12:17:29Z")
quick question, which version are you using? any chance you can try the same thing with 0.20.5? simon On Friday, March 1, 2013 12:46:22 PM UTC+1, amos.wood wrote: > Simon, > > It makes sense to explicitly call the refresh. If you look at my gist, I > am doing that with the 3rd record. When I explictly call the refresh, the > search always return. > > It also makes sense that a refresh that is triggered intravenously through > the use of the refresh\_interval could actually take longer than the 500ms > that it is triggered with. However in my test, I have the refresh interval > set to 500ms and I am pausing for 5 seconds and the search is not coming > back. I have actually seen the same thing happen when pausing for 10 > seconds. Then after waiting for 5 or 10 seconds, searching, and then > immediately searching again and then getting the results is not right. I > could see the search not returning in 600ms, but 10s later on a new index > which I just created or with no other things going on with the cluster, it > seems like something is wrong. > > Note: I am only seeing the problem on a cluster which can support a > replicas count of 2 for the index. It seems like in the round-robin search > approach, that the first /shard/replica that it is searching hasn't been > indexed while the second searched shard/replica does have it indexed. > After 10 seconds though, I don't know why everything wouldn't be indexed on > the cluster though. > > Amos > > On Friday, March 1, 2013 3:22:05 AM UTC-6, simonw wrote: > > > On Friday, March 1, 2013 4:24:11 AM UTC+1, amos.wood wrote: > > > > > Here is my search use case which seems to be failing periodically: > > > > > > 1. Create an index with a 500ms refresh interval > > > 2. Write a document to an index > > > 3. Pause for between 500ms and 5s > > > 4. Attempt to search for a value in 1 field > > > 5. Attempt to search again for the same value in the same field > > > > > > In my attempt to search, sometimes I am getting back no results on the > > > first try with the second search attempt succeeding. Other times, I am > > > getting back results on both searches. Even more strange, sometimes I get > > > back a result on the first search but not on the second search. Does > > > anyone know what may be going on? > > > > Hey, > > > > the refresh\_interval is not guaranteeing to make results visible 500ms > > after you write the document to the index. it depends on many things like > > how long it takes the do the refresh etc. so the interval can be higher in > > practice. If you want to run this testcase you should make sure to call > > refresh explicitly via the rest API to make sure that everything has been > > refreshed. > > > > simon > > > > > I have a gist script which will simulate the issue. I could run this > > > gist on our production environment cluster and see the issue happen about > > > every other run. When I run it on our sandbox environment, I cannot > > > reproduce the issue. > > > > > > [ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub](https://gist.github.com/anonymous/78e12092e648e96c20f0) > > > > > > _Production Cluster Configuration:_ > > > > > > Elasticsearch Version 0.19.4 > > > > > > 1 Master Non Data Node > > > 1 Master Data Node > > > 6 Data Nodes > > > > > > All VM servers with 4 CPUs with 18GB memory. > > > > > > _Sandbox Cluster Configuration:_ > > > Elasticsearch Version 0.19.4 > > > > > > 2 Master Data Nodes > > > > > > Both VM servers with 2 CPUs with 8GB memory. > > On Friday, March 1, 2013 3:22:05 AM UTC-6, simonw wrote: > > > On Friday, March 1, 2013 4:24:11 AM UTC+1, amos.wood wrote: > > > > > Here is my search use case which seems to be failing periodically: > > > > > > 1. Create an index with a 500ms refresh interval > > > 2. Write a document to an index > > > 3. Pause for between 500ms and 5s > > > 4. Attempt to search for a value in 1 field > > > 5. Attempt to search again for the same value in the same field > > > > > > In my attempt to search, sometimes I am getting back no results on the > > > first try with the second search attempt succeeding. Other times, I am > > > getting back results on both searches. Even more strange, sometimes I get > > > back a result on the first search but not on the second search. Does > > > anyone know what may be going on? > > > > Hey, > > > > the refresh\_interval is not guaranteeing to make results visible 500ms > > after you write the document to the index. it depends on many things like > > how long it takes the do the refresh etc. so the interval can be higher in > > practice. If you want to run this testcase you should make sure to call > > refresh explicitly via the rest API to make sure that everything has been > > refreshed. > > > > simon > > > > > I have a gist script which will simulate the issue. I could run this > > > gist on our production environment cluster and see the issue happen about > > > every other run. When I run it on our sandbox environment, I cannot > > > reproduce the issue. > > > > > > [ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub](https://gist.github.com/anonymous/78e12092e648e96c20f0) > > > > > > _Production Cluster Configuration:_ > > > > > > Elasticsearch Version 0.19.4 > > > > > > 1 Master Non Data Node > > > 1 Master Data Node > > > 6 Data Nodes > > > > > > All VM servers with 4 CPUs with 18GB memory. > > > > > > _Sandbox Cluster Configuration:_ > > > Elasticsearch Version 0.19.4 > > > > > > 2 Master Data Nodes > > > > > > Both VM servers with 2 CPUs with 8GB memory. > > On Friday, March 1, 2013 3:22:05 AM UTC-6, simonw wrote: > > > On Friday, March 1, 2013 4:24:11 AM UTC+1, amos.wood wrote: > > > > > Here is my search use case which seems to be failing periodically: > > > > > > 1. Create an index with a 500ms refresh interval > > > 2. Write a document to an index > > > 3. Pause for between 500ms and 5s > > > 4. Attempt to search for a value in 1 field > > > 5. Attempt to search again for the same value in the same field > > > > > > In my attempt to search, sometimes I am getting back no results on the > > > first try with the second search attempt succeeding. Other times, I am > > > getting back results on both searches. Even more strange, sometimes I get > > > back a result on the first search but not on the second search. Does > > > anyone know what may be going on? > > > > Hey, > > > > the refresh\_interval is not guaranteeing to make results visible 500ms > > after you write the document to the index. it depends on many things like > > how long it takes the do the refresh etc. so the interval can be higher in > > practice. If you want to run this testcase you should make sure to call > > refresh explicitly via the rest API to make sure that everything has been > > refreshed. > > > > simon > > > > > I have a gist script which will simulate the issue. I could run this > > > gist on our production environment cluster and see the issue happen about > > > every other run. When I run it on our sandbox environment, I cannot > > > reproduce the issue. > > > > > > [ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub](https://gist.github.com/anonymous/78e12092e648e96c20f0) > > > > > > _Production Cluster Configuration:_ > > > > > > Elasticsearch Version 0.19.4 > > > > > > 1 Master Non Data Node > > > 1 Master Data Node > > > 6 Data Nodes > > > > > > All VM servers with 4 CPUs with 18GB memory. > > > > > > _Sandbox Cluster Configuration:_ > > > Elasticsearch Version 0.19.4 > > > > > > 2 Master Data Nodes > > > > > > Both VM servers with 2 CPUs with 8GB memory. > > On Friday, March 1, 2013 3:22:05 AM UTC-6, simonw wrote: > > > On Friday, March 1, 2013 4:24:11 AM UTC+1, amos.wood wrote: > > > > > Here is my search use case which seems to be failing periodically: > > > > > > 1. Create an index with a 500ms refresh interval > > > 2. Write a document to an index > > > 3. Pause for between 500ms and 5s > > > 4. Attempt to search for a value in 1 field > > > 5. Attempt to search again for the same value in the same field > > > > > > In my attempt to search, sometimes I am getting back no results on the > > > first try with the second search attempt succeeding. Other times, I am > > > getting back results on both searches. Even more strange, sometimes I get > > > back a result on the first search but not on the second search. Does > > > anyone know what may be going on? > > > > Hey, > > > > the refresh\_interval is not guaranteeing to make results visible 500ms > > after you write the document to the index. it depends on many things like > > how long it takes the do the refresh etc. so the interval can be higher in > > practice. If you want to run this testcase you should make sure to call > > refresh explicitly via the rest API to make sure that everything has been > > refreshed. > > > > simon > > > > > I have a gist script which will simulate the issue. I could run this > > > gist on our production environment cluster and see the issue happen about > > > every other run. When I run it on our sandbox environment, I cannot > > > reproduce the issue. > > > > > > [ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub](https://gist.github.com/anonymous/78e12092e648e96c20f0) > > > > > > _Production Cluster Configuration:_ > > > > > > Elasticsearch Version 0.19.4 > > > > > > 1 Master Non Data Node > > > 1 Master Data Node > > > 6 Data Nodes > > > > > > All VM servers with 4 CPUs with 18GB memory. > > > > > > _Sandbox Cluster Configuration:_ > > > Elasticsearch Version 0.19.4 > > > > > > 2 Master Data Nodes > > > > > > Both VM servers with 2 CPUs with 8GB memory. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [March 1, 2013, 12:18pm UTC](https://discuss.elastic.co/t/search-periodically-failing-even-after-many-refresh-intervals/10962/5 "2013-03-01T12:18:26Z")
On Thu, 2013-02-28 at 19:24 -0800, amos.wood wrote: > Here is my search use case which seems to be failing periodically: > 1. Create an index with a 500ms refresh interval > 2. Write a document to an index > 3. Pause for between 500ms and 5s > 4. Attempt to search for a value in 1 field > 5. Attempt to search again for the same value in the same field I'm wondering if your cluster is green and in sync? Take a look at the cluster health, make sure all your nodes are in there. Try doing multiple /\_count requests, see if you get back different results when hitting different nodes. Perhaps bounce your cluster clint > In my attempt to search, sometimes I am getting back no results on the > first try with the second search attempt succeeding. Other times, I > am getting back results on both searches. Even more strange, > sometimes I get back a result on the first search but not on the > second search. Does anyone know what may be going on? > > I have a gist script which will simulate the issue. I could run this > gist on our production environment cluster and see the issue happen > about every other run. When I run it on our sandbox environment, I > cannot reproduce the issue. > > [ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub](https://gist.github.com/anonymous/78e12092e648e96c20f0) > > Production Cluster Configuration: > > Elasticsearch Version 0.19.4 > > 1 Master Non Data Node > 1 Master Data Node > 6 Data Nodes > > All VM servers with 4 CPUs with 18GB memory. > > Sandbox Cluster Configuration: > > Elasticsearch Version 0.19.4 > > 2 Master Data Nodes > > Both VM servers with 2 CPUs with 8GB memory. > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![amos\_wood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amos_wood/32/1880_2.png) [@amos\_wood](https://discuss.elastic.co/u/amos_wood)\ **Post date:** [March 1, 2013, 1:19pm UTC](https://discuss.elastic.co/t/search-periodically-failing-even-after-many-refresh-intervals/10962/6 "2013-03-01T13:19:41Z")
We are using 0.19.4. I could upgrade to 0.20.5 and try the same thing, but that would take quite awhile to do since there are 8 nodes in the cluster. As for the cluster health, I get the following report: { - cluster\_name: indextest - status: green - timed\_out: false - number\_of\_nodes: 8 - number\_of\_data\_nodes: 7 - active\_primary\_shards: 134 - active\_shards: 430 - relocating\_shards: 0 - initializing\_shards: 0 - unassigned\_shards: 0 } I have verified that the 8 nodes IP address in the elasticsearch.yml file as well: bootstrap: mlockall: true path: logs: /index/elasticsearch/logs data: /index/elasticsearch/data gateway: type: local recover\_after\_nodes: 4 recover\_after\_time: 5m expected\_nodes: 8 discovery: zen: ping\_timeout: 3s minimum\_master\_nodes: 2 ping: multicast: enabled: false unicast: enabled: true hosts: [172.16.47.133 ,172.16.47.134 ,172.16.47.135 ,172.16.47.157 ,172.16.46.88 ,172.16.47.53 ,172.16.47.66 ,172.16.47.82] fd: ping\_interval: 1s ping\_timeout: 30s ping\_retries: 3 script: default\_lang: mvel cluster: name: indextest node: name: indextest1 master: true client: false data: false action: auto\_create\_index: false disable\_delete\_all\_indices: true get: realtime: true indices: query: bool: max\_clause\_count : 15000 On Friday, March 1, 2013 6:18:26 AM UTC-6, Clinton Gormley wrote: > On Thu, 2013-02-28 at 19:24 -0800, amos.wood wrote: > > > Here is my search use case which seems to be failing periodically: > > 1. Create an index with a 500ms refresh interval > > 2. Write a document to an index > > 3. Pause for between 500ms and 5s > > 4. Attempt to search for a value in 1 field > > 5. Attempt to search again for the same value in the same field > > I'm wondering if your cluster is green and in sync? > > Take a look at the cluster health, make sure all your nodes are in > there. > > Try doing multiple /\_count requests, see if you get back different > results when hitting different nodes. > > Perhaps bounce your cluster > > clint > > > In my attempt to search, sometimes I am getting back no results on the > > first try with the second search attempt succeeding. Other times, I > > am getting back results on both searches. Even more strange, > > sometimes I get back a result on the first search but not on the > > second search. Does anyone know what may be going on? > > > > I have a gist script which will simulate the issue. I could run this > > gist on our production environment cluster and see the issue happen > > about every other run. When I run it on our sandbox environment, I > > cannot reproduce the issue. > > > > [ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub](https://gist.github.com/anonymous/78e12092e648e96c20f0) > > > > Production Cluster Configuration: > > > > Elasticsearch Version 0.19.4 > > > > 1 Master Non Data Node > > 1 Master Data Node > > 6 Data Nodes > > > > All VM servers with 4 CPUs with 18GB memory. > > > > Sandbox Cluster Configuration: > > > > Elasticsearch Version 0.19.4 > > > > 2 Master Data Nodes > > > > Both VM servers with 2 CPUs with 8GB memory. > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![amos\_wood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amos_wood/32/1880_2.png) [@amos\_wood](https://discuss.elastic.co/u/amos_wood)\ **Post date:** [March 1, 2013, 3:15pm UTC](https://discuss.elastic.co/t/search-periodically-failing-even-after-many-refresh-intervals/10962/7 "2013-03-01T15:15:02Z")
After upgrading I didn't see the issue. Also, I then went back to version 0.19.4, and I am not seeing the issue any more either. It may have been a restart issue or a incomplete configuration issue on the cluster which resolved when I confirmed all elasticsearch.yml files were correct and rebooted the cluster. I can say now that the cluster is performing at great speeds and I am not seeing any latency issues when searching immediately after the pause of 500ms. Thanks for your help. On Friday, March 1, 2013 7:19:41 AM UTC-6, amos.wood wrote: > We are using 0.19.4. I could upgrade to 0.20.5 and try the same thing, > but that would take quite awhile to do since there are 8 nodes in the > cluster. > > As for the cluster health, I get the following report: > > { > > - cluster\_name: indextest > - status: green > - timed\_out: false > - number\_of\_nodes: 8 > - number\_of\_data\_nodes: 7 > - active\_primary\_shards: 134 > - active\_shards: 430 > - relocating\_shards: 0 > - initializing\_shards: 0 > - unassigned\_shards: 0 > > } > > I have verified that the 8 nodes IP address in the elasticsearch.yml file > as well: > > bootstrap: > mlockall: true > path: > logs: /index/elasticsearch/logs > data: /index/elasticsearch/data > gateway: > type: local > recover\_after\_nodes: 4 > recover\_after\_time: 5m > expected\_nodes: 8 > discovery: > zen: > ping\_timeout: 3s > minimum\_master\_nodes: 2 > ping: > multicast: > enabled: false > unicast: > enabled: true > hosts: [172.16.47.133 > ,172.16.47.134 > ,172.16.47.135 > ,172.16.47.157 > ,172.16.46.88 > ,172.16.47.53 > ,172.16.47.66 > ,172.16.47.82] > fd: > ping\_interval: 1s > ping\_timeout: 30s > ping\_retries: 3 > script: > default\_lang: mvel > cluster: > name: indextest > node: > name: indextest1 > master: true > client: false > data: false > action: > auto\_create\_index: false > disable\_delete\_all\_indices: true > get: > realtime: true > indices: > query: > bool: > max\_clause\_count : 15000 > > On Friday, March 1, 2013 6:18:26 AM UTC-6, Clinton Gormley wrote: > > > On Thu, 2013-02-28 at 19:24 -0800, amos.wood wrote: > > > > > Here is my search use case which seems to be failing periodically: > > > 1. Create an index with a 500ms refresh interval > > > 2. Write a document to an index > > > 3. Pause for between 500ms and 5s > > > 4. Attempt to search for a value in 1 field > > > 5. Attempt to search again for the same value in the same field > > > > I'm wondering if your cluster is green and in sync? > > > > Take a look at the cluster health, make sure all your nodes are in > > there. > > > > Try doing multiple /\_count requests, see if you get back different > > results when hitting different nodes. > > > > Perhaps bounce your cluster > > > > clint > > > > > In my attempt to search, sometimes I am getting back no results on the > > > first try with the second search attempt succeeding. Other times, I > > > am getting back results on both searches. Even more strange, > > > sometimes I get back a result on the first search but not on the > > > second search. Does anyone know what may be going on? > > > > > > I have a gist script which will simulate the issue. I could run this > > > gist on our production environment cluster and see the issue happen > > > about every other run. When I run it on our sandbox environment, I > > > cannot reproduce the issue. > > > > > > [ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub](https://gist.github.com/anonymous/78e12092e648e96c20f0) > > > > > > Production Cluster Configuration: > > > > > > Elasticsearch Version 0.19.4 > > > > > > 1 Master Non Data Node > > > 1 Master Data Node > > > 6 Data Nodes > > > > > > All VM servers with 4 CPUs with 18GB memory. > > > > > > Sandbox Cluster Configuration: > > > > > > Elasticsearch Version 0.19.4 > > > > > > 2 Master Data Nodes > > > > > > Both VM servers with 2 CPUs with 8GB memory. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![simonw\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonw_2/32/1130_2.png) [@simonw\_2](https://discuss.elastic.co/u/simonw_2)\ **Post date:** [March 1, 2013, 4:12pm UTC](https://discuss.elastic.co/t/search-periodically-failing-even-after-many-refresh-intervals/10962/8 "2013-03-01T16:12:01Z")
thanks for reporting back! On Friday, March 1, 2013 4:15:02 PM UTC+1, amos.wood wrote: > After upgrading I didn't see the issue. Also, I then went back to version > 0.19.4, and I am not seeing the issue any more either. It may have been a > restart issue or a incomplete configuration issue on the cluster which > resolved when I confirmed all elasticsearch.yml files were correct and > rebooted the cluster. I can say now that the cluster is performing at > great speeds and I am not seeing any latency issues when searching > immediately after the pause of 500ms. > > Thanks for your help. > > On Friday, March 1, 2013 7:19:41 AM UTC-6, amos.wood wrote: > > > We are using 0.19.4. I could upgrade to 0.20.5 and try the same thing, > > but that would take quite awhile to do since there are 8 nodes in the > > cluster. > > > > As for the cluster health, I get the following report: > > > > { > > > > - cluster\_name: indextest > > - status: green > > - timed\_out: false > > - number\_of\_nodes: 8 > > - number\_of\_data\_nodes: 7 > > - active\_primary\_shards: 134 > > - active\_shards: 430 > > - relocating\_shards: 0 > > - initializing\_shards: 0 > > - unassigned\_shards: 0 > > > > } > > > > I have verified that the 8 nodes IP address in the elasticsearch.yml file > > as well: > > > > bootstrap: > > mlockall: true > > path: > > logs: /index/elasticsearch/logs > > data: /index/elasticsearch/data > > gateway: > > type: local > > recover\_after\_nodes: 4 > > recover\_after\_time: 5m > > expected\_nodes: 8 > > discovery: > > zen: > > ping\_timeout: 3s > > minimum\_master\_nodes: 2 > > ping: > > multicast: > > enabled: false > > unicast: > > enabled: true > > hosts: [172.16.47.133 > > ,172.16.47.134 > > ,172.16.47.135 > > ,172.16.47.157 > > ,172.16.46.88 > > ,172.16.47.53 > > ,172.16.47.66 > > ,172.16.47.82] > > fd: > > ping\_interval: 1s > > ping\_timeout: 30s > > ping\_retries: 3 > > script: > > default\_lang: mvel > > cluster: > > name: indextest > > node: > > name: indextest1 > > master: true > > client: false > > data: false > > action: > > auto\_create\_index: false > > disable\_delete\_all\_indices: true > > get: > > realtime: true > > indices: > > query: > > bool: > > max\_clause\_count : 15000 > > > > On Friday, March 1, 2013 6:18:26 AM UTC-6, Clinton Gormley wrote: > > > > > On Thu, 2013-02-28 at 19:24 -0800, amos.wood wrote: > > > > > > > Here is my search use case which seems to be failing periodically: > > > > 1. Create an index with a 500ms refresh interval > > > > 2. Write a document to an index > > > > 3. Pause for between 500ms and 5s > > > > 4. Attempt to search for a value in 1 field > > > > 5. Attempt to search again for the same value in the same field > > > > > > I'm wondering if your cluster is green and in sync? > > > > > > Take a look at the cluster health, make sure all your nodes are in > > > there. > > > > > > Try doing multiple /\_count requests, see if you get back different > > > results when hitting different nodes. > > > > > > Perhaps bounce your cluster > > > > > > clint > > > > > > > In my attempt to search, sometimes I am getting back no results on the > > > > first try with the second search attempt succeeding. Other times, I > > > > am getting back results on both searches. Even more strange, > > > > sometimes I get back a result on the first search but not on the > > > > second search. Does anyone know what may be going on? > > > > > > > > I have a gist script which will simulate the issue. I could run this > > > > gist on our production environment cluster and see the issue happen > > > > about every other run. When I run it on our sandbox environment, I > > > > cannot reproduce the issue. > > > > > > > > [ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub](https://gist.github.com/anonymous/78e12092e648e96c20f0) > > > > > > > > Production Cluster Configuration: > > > > > > > > Elasticsearch Version 0.19.4 > > > > > > > > 1 Master Non Data Node > > > > 1 Master Data Node > > > > 6 Data Nodes > > > > > > > > All VM servers with 4 CPUs with 18GB memory. > > > > > > > > Sandbox Cluster Configuration: > > > > > > > > Elasticsearch Version 0.19.4 > > > > > > > > 2 Master Data Nodes > > > > > > > > Both VM servers with 2 CPUs with 8GB memory. > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:48am UTC](https://discuss.elastic.co/t/search-periodically-failing-even-after-many-refresh-intervals/10962/9 "2017-07-06T02:48:49Z")