Search Periodically Failing ... Even After Many Refresh Intervals

Elastic Stack Elasticsearch
1

Here is my search use case which seems to be failing periodically:

  1. Create an index with a 500ms refresh interval
  2. Write a document to an index
  3. Pause for between 500ms and 5s
  4. Attempt to search for a value in 1 field
  5. Attempt to search again for the same value in the same field

In my attempt to search, sometimes I am getting back no results on the
first try with the second search attempt succeeding. Other times, I am
getting back results on both searches. Even more strange, sometimes I get
back a result on the first search but not on the second search. Does
anyone know what may be going on?

I have a gist script which will simulate the issue. I could run this gist
on our production environment cluster and see the issue happen about every
other run. When I run it on our sandbox environment, I cannot reproduce
the issue.

Production Cluster Configuration:

ElasticSearch Version 0.19.4

1 Master Non Data Node
1 Master Data Node
6 Data Nodes

All VM servers with 4 CPUs with 18GB memory.

Sandbox Cluster Configuration:
ElasticSearch Version 0.19.4

2 Master Data Nodes

Both VM servers with 2 CPUs with 8GB memory.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

On Friday, March 1, 2013 4:24:11 AM UTC+1, amos.wood wrote:

Here is my search use case which seems to be failing periodically:

  1. Create an index with a 500ms refresh interval
  2. Write a document to an index
  3. Pause for between 500ms and 5s
  4. Attempt to search for a value in 1 field
  5. Attempt to search again for the same value in the same field

In my attempt to search, sometimes I am getting back no results on the
first try with the second search attempt succeeding. Other times, I am
getting back results on both searches. Even more strange, sometimes I get
back a result on the first search but not on the second search. Does
anyone know what may be going on?

Hey,

the refresh_interval is not guaranteeing to make results visible 500ms
after you write the document to the index. it depends on many things like
how long it takes the do the refresh etc. so the interval can be higher in
practice. If you want to run this testcase you should make sure to call
refresh explicitly via the rest API to make sure that everything has been
refreshed.

simon

I have a gist script which will simulate the issue. I could run this gist
on our production environment cluster and see the issue happen about every
other run. When I run it on our sandbox environment, I cannot reproduce
the issue.

ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub

Production Cluster Configuration:

Elasticsearch Version 0.19.4

1 Master Non Data Node
1 Master Data Node
6 Data Nodes

All VM servers with 4 CPUs with 18GB memory.

Sandbox Cluster Configuration:
Elasticsearch Version 0.19.4

2 Master Data Nodes

Both VM servers with 2 CPUs with 8GB memory.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Simon,

It makes sense to explicitly call the refresh. If you look at my gist, I
am doing that with the 3rd record. When I explictly call the refresh, the
search always return.

It also makes sense that a refresh that is triggered intravenously through
the use of the refresh_interval could actually take longer than the 500ms
that it is triggered with. However in my test, I have the refresh interval
set to 500ms and I am pausing for 5 seconds and the search is not coming
back. I have actually seen the same thing happen when pausing for 10
seconds. Then after waiting for 5 or 10 seconds, searching, and then
immediately searching again and then getting the results is not right. I
could see the search not returning in 600ms, but 10s later on a new index
which I just created or with no other things going on with the cluster, it
seems like something is wrong.

Note: I am only seeing the problem on a cluster which can support a
replicas count of 2 for the index. It seems like in the round-robin search
approach, that the first /shard/replica that it is searching hasn't been
indexed while the second searched shard/replica does have it indexed.
After 10 seconds though, I don't know why everything wouldn't be indexed on
the cluster though.

Amos

On Friday, March 1, 2013 3:22:05 AM UTC-6, simonw wrote:

On Friday, March 1, 2013 4:24:11 AM UTC+1, amos.wood wrote:

Here is my search use case which seems to be failing periodically:

  1. Create an index with a 500ms refresh interval
  2. Write a document to an index
  3. Pause for between 500ms and 5s
  4. Attempt to search for a value in 1 field
  5. Attempt to search again for the same value in the same field

In my attempt to search, sometimes I am getting back no results on the
first try with the second search attempt succeeding. Other times, I am
getting back results on both searches. Even more strange, sometimes I get
back a result on the first search but not on the second search. Does
anyone know what may be going on?

Hey,

the refresh_interval is not guaranteeing to make results visible 500ms
after you write the document to the index. it depends on many things like
how long it takes the do the refresh etc. so the interval can be higher in
practice. If you want to run this testcase you should make sure to call
refresh explicitly via the rest API to make sure that everything has been
refreshed.

simon

I have a gist script which will simulate the issue. I could run this
gist on our production environment cluster and see the issue happen about
every other run. When I run it on our sandbox environment, I cannot
reproduce the issue.

ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub

Production Cluster Configuration:

Elasticsearch Version 0.19.4

1 Master Non Data Node
1 Master Data Node
6 Data Nodes

All VM servers with 4 CPUs with 18GB memory.

Sandbox Cluster Configuration:
Elasticsearch Version 0.19.4

2 Master Data Nodes

Both VM servers with 2 CPUs with 8GB memory.

On Friday, March 1, 2013 3:22:05 AM UTC-6, simonw wrote:

On Friday, March 1, 2013 4:24:11 AM UTC+1, amos.wood wrote:

Here is my search use case which seems to be failing periodically:

  1. Create an index with a 500ms refresh interval
  2. Write a document to an index
  3. Pause for between 500ms and 5s
  4. Attempt to search for a value in 1 field
  5. Attempt to search again for the same value in the same field

In my attempt to search, sometimes I am getting back no results on the
first try with the second search attempt succeeding. Other times, I am
getting back results on both searches. Even more strange, sometimes I get
back a result on the first search but not on the second search. Does
anyone know what may be going on?

Hey,

the refresh_interval is not guaranteeing to make results visible 500ms
after you write the document to the index. it depends on many things like
how long it takes the do the refresh etc. so the interval can be higher in
practice. If you want to run this testcase you should make sure to call
refresh explicitly via the rest API to make sure that everything has been
refreshed.

simon

I have a gist script which will simulate the issue. I could run this
gist on our production environment cluster and see the issue happen about
every other run. When I run it on our sandbox environment, I cannot
reproduce the issue.

ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub

Production Cluster Configuration:

Elasticsearch Version 0.19.4

1 Master Non Data Node
1 Master Data Node
6 Data Nodes

All VM servers with 4 CPUs with 18GB memory.

Sandbox Cluster Configuration:
Elasticsearch Version 0.19.4

2 Master Data Nodes

Both VM servers with 2 CPUs with 8GB memory.

On Friday, March 1, 2013 3:22:05 AM UTC-6, simonw wrote:

On Friday, March 1, 2013 4:24:11 AM UTC+1, amos.wood wrote:

Here is my search use case which seems to be failing periodically:

  1. Create an index with a 500ms refresh interval
  2. Write a document to an index
  3. Pause for between 500ms and 5s
  4. Attempt to search for a value in 1 field
  5. Attempt to search again for the same value in the same field

In my attempt to search, sometimes I am getting back no results on the
first try with the second search attempt succeeding. Other times, I am
getting back results on both searches. Even more strange, sometimes I get
back a result on the first search but not on the second search. Does
anyone know what may be going on?

Hey,

the refresh_interval is not guaranteeing to make results visible 500ms
after you write the document to the index. it depends on many things like
how long it takes the do the refresh etc. so the interval can be higher in
practice. If you want to run this testcase you should make sure to call
refresh explicitly via the rest API to make sure that everything has been
refreshed.

simon

I have a gist script which will simulate the issue. I could run this
gist on our production environment cluster and see the issue happen about
every other run. When I run it on our sandbox environment, I cannot
reproduce the issue.

ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub

Production Cluster Configuration:

Elasticsearch Version 0.19.4

1 Master Non Data Node
1 Master Data Node
6 Data Nodes

All VM servers with 4 CPUs with 18GB memory.

Sandbox Cluster Configuration:
Elasticsearch Version 0.19.4

2 Master Data Nodes

Both VM servers with 2 CPUs with 8GB memory.

On Friday, March 1, 2013 3:22:05 AM UTC-6, simonw wrote:

On Friday, March 1, 2013 4:24:11 AM UTC+1, amos.wood wrote:

Here is my search use case which seems to be failing periodically:

  1. Create an index with a 500ms refresh interval
  2. Write a document to an index
  3. Pause for between 500ms and 5s
  4. Attempt to search for a value in 1 field
  5. Attempt to search again for the same value in the same field

In my attempt to search, sometimes I am getting back no results on the
first try with the second search attempt succeeding. Other times, I am
getting back results on both searches. Even more strange, sometimes I get
back a result on the first search but not on the second search. Does
anyone know what may be going on?

Hey,

the refresh_interval is not guaranteeing to make results visible 500ms
after you write the document to the index. it depends on many things like
how long it takes the do the refresh etc. so the interval can be higher in
practice. If you want to run this testcase you should make sure to call
refresh explicitly via the rest API to make sure that everything has been
refreshed.

simon

I have a gist script which will simulate the issue. I could run this
gist on our production environment cluster and see the issue happen about
every other run. When I run it on our sandbox environment, I cannot
reproduce the issue.

ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub

Production Cluster Configuration:

Elasticsearch Version 0.19.4

1 Master Non Data Node
1 Master Data Node
6 Data Nodes

All VM servers with 4 CPUs with 18GB memory.

Sandbox Cluster Configuration:
Elasticsearch Version 0.19.4

2 Master Data Nodes

Both VM servers with 2 CPUs with 8GB memory.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

quick question, which version are you using? any chance you can try the
same thing with 0.20.5?

simon

On Friday, March 1, 2013 12:46:22 PM UTC+1, amos.wood wrote:

Simon,

It makes sense to explicitly call the refresh. If you look at my gist, I
am doing that with the 3rd record. When I explictly call the refresh, the
search always return.

It also makes sense that a refresh that is triggered intravenously through
the use of the refresh_interval could actually take longer than the 500ms
that it is triggered with. However in my test, I have the refresh interval
set to 500ms and I am pausing for 5 seconds and the search is not coming
back. I have actually seen the same thing happen when pausing for 10
seconds. Then after waiting for 5 or 10 seconds, searching, and then
immediately searching again and then getting the results is not right. I
could see the search not returning in 600ms, but 10s later on a new index
which I just created or with no other things going on with the cluster, it
seems like something is wrong.

Note: I am only seeing the problem on a cluster which can support a
replicas count of 2 for the index. It seems like in the round-robin search
approach, that the first /shard/replica that it is searching hasn't been
indexed while the second searched shard/replica does have it indexed.
After 10 seconds though, I don't know why everything wouldn't be indexed on
the cluster though.

Amos

On Friday, March 1, 2013 3:22:05 AM UTC-6, simonw wrote:

On Friday, March 1, 2013 4:24:11 AM UTC+1, amos.wood wrote:

Here is my search use case which seems to be failing periodically:

  1. Create an index with a 500ms refresh interval
  2. Write a document to an index
  3. Pause for between 500ms and 5s
  4. Attempt to search for a value in 1 field
  5. Attempt to search again for the same value in the same field

In my attempt to search, sometimes I am getting back no results on the
first try with the second search attempt succeeding. Other times, I am
getting back results on both searches. Even more strange, sometimes I get
back a result on the first search but not on the second search. Does
anyone know what may be going on?

Hey,

the refresh_interval is not guaranteeing to make results visible 500ms
after you write the document to the index. it depends on many things like
how long it takes the do the refresh etc. so the interval can be higher in
practice. If you want to run this testcase you should make sure to call
refresh explicitly via the rest API to make sure that everything has been
refreshed.

simon

I have a gist script which will simulate the issue. I could run this
gist on our production environment cluster and see the issue happen about
every other run. When I run it on our sandbox environment, I cannot
reproduce the issue.

ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub

Production Cluster Configuration:

Elasticsearch Version 0.19.4

1 Master Non Data Node
1 Master Data Node
6 Data Nodes

All VM servers with 4 CPUs with 18GB memory.

Sandbox Cluster Configuration:
Elasticsearch Version 0.19.4

2 Master Data Nodes

Both VM servers with 2 CPUs with 8GB memory.

On Friday, March 1, 2013 3:22:05 AM UTC-6, simonw wrote:

On Friday, March 1, 2013 4:24:11 AM UTC+1, amos.wood wrote:

Here is my search use case which seems to be failing periodically:

  1. Create an index with a 500ms refresh interval
  2. Write a document to an index
  3. Pause for between 500ms and 5s
  4. Attempt to search for a value in 1 field
  5. Attempt to search again for the same value in the same field

In my attempt to search, sometimes I am getting back no results on the
first try with the second search attempt succeeding. Other times, I am
getting back results on both searches. Even more strange, sometimes I get
back a result on the first search but not on the second search. Does
anyone know what may be going on?

Hey,

the refresh_interval is not guaranteeing to make results visible 500ms
after you write the document to the index. it depends on many things like
how long it takes the do the refresh etc. so the interval can be higher in
practice. If you want to run this testcase you should make sure to call
refresh explicitly via the rest API to make sure that everything has been
refreshed.

simon

I have a gist script which will simulate the issue. I could run this
gist on our production environment cluster and see the issue happen about
every other run. When I run it on our sandbox environment, I cannot
reproduce the issue.

ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub

Production Cluster Configuration:

Elasticsearch Version 0.19.4

1 Master Non Data Node
1 Master Data Node
6 Data Nodes

All VM servers with 4 CPUs with 18GB memory.

Sandbox Cluster Configuration:
Elasticsearch Version 0.19.4

2 Master Data Nodes

Both VM servers with 2 CPUs with 8GB memory.

On Friday, March 1, 2013 3:22:05 AM UTC-6, simonw wrote:

On Friday, March 1, 2013 4:24:11 AM UTC+1, amos.wood wrote:

Here is my search use case which seems to be failing periodically:

  1. Create an index with a 500ms refresh interval
  2. Write a document to an index
  3. Pause for between 500ms and 5s
  4. Attempt to search for a value in 1 field
  5. Attempt to search again for the same value in the same field

In my attempt to search, sometimes I am getting back no results on the
first try with the second search attempt succeeding. Other times, I am
getting back results on both searches. Even more strange, sometimes I get
back a result on the first search but not on the second search. Does
anyone know what may be going on?

Hey,

the refresh_interval is not guaranteeing to make results visible 500ms
after you write the document to the index. it depends on many things like
how long it takes the do the refresh etc. so the interval can be higher in
practice. If you want to run this testcase you should make sure to call
refresh explicitly via the rest API to make sure that everything has been
refreshed.

simon

I have a gist script which will simulate the issue. I could run this
gist on our production environment cluster and see the issue happen about
every other run. When I run it on our sandbox environment, I cannot
reproduce the issue.

ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub

Production Cluster Configuration:

Elasticsearch Version 0.19.4

1 Master Non Data Node
1 Master Data Node
6 Data Nodes

All VM servers with 4 CPUs with 18GB memory.

Sandbox Cluster Configuration:
Elasticsearch Version 0.19.4

2 Master Data Nodes

Both VM servers with 2 CPUs with 8GB memory.

On Friday, March 1, 2013 3:22:05 AM UTC-6, simonw wrote:

On Friday, March 1, 2013 4:24:11 AM UTC+1, amos.wood wrote:

Here is my search use case which seems to be failing periodically:

  1. Create an index with a 500ms refresh interval
  2. Write a document to an index
  3. Pause for between 500ms and 5s
  4. Attempt to search for a value in 1 field
  5. Attempt to search again for the same value in the same field

In my attempt to search, sometimes I am getting back no results on the
first try with the second search attempt succeeding. Other times, I am
getting back results on both searches. Even more strange, sometimes I get
back a result on the first search but not on the second search. Does
anyone know what may be going on?

Hey,

the refresh_interval is not guaranteeing to make results visible 500ms
after you write the document to the index. it depends on many things like
how long it takes the do the refresh etc. so the interval can be higher in
practice. If you want to run this testcase you should make sure to call
refresh explicitly via the rest API to make sure that everything has been
refreshed.

simon

I have a gist script which will simulate the issue. I could run this
gist on our production environment cluster and see the issue happen about
every other run. When I run it on our sandbox environment, I cannot
reproduce the issue.

ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub

Production Cluster Configuration:

Elasticsearch Version 0.19.4

1 Master Non Data Node
1 Master Data Node
6 Data Nodes

All VM servers with 4 CPUs with 18GB memory.

Sandbox Cluster Configuration:
Elasticsearch Version 0.19.4

2 Master Data Nodes

Both VM servers with 2 CPUs with 8GB memory.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

On Thu, 2013-02-28 at 19:24 -0800, amos.wood wrote:

Here is my search use case which seems to be failing periodically:
1. Create an index with a 500ms refresh interval
2. Write a document to an index
3. Pause for between 500ms and 5s
4. Attempt to search for a value in 1 field
5. Attempt to search again for the same value in the same field

I'm wondering if your cluster is green and in sync?

Take a look at the cluster health, make sure all your nodes are in
there.

Try doing multiple /_count requests, see if you get back different
results when hitting different nodes.

Perhaps bounce your cluster

clint

In my attempt to search, sometimes I am getting back no results on the
first try with the second search attempt succeeding. Other times, I
am getting back results on both searches. Even more strange,
sometimes I get back a result on the first search but not on the
second search. Does anyone know what may be going on?

I have a gist script which will simulate the issue. I could run this
gist on our production environment cluster and see the issue happen
about every other run. When I run it on our sandbox environment, I
cannot reproduce the issue.

ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub

Production Cluster Configuration:

Elasticsearch Version 0.19.4

1 Master Non Data Node
1 Master Data Node
6 Data Nodes

All VM servers with 4 CPUs with 18GB memory.

Sandbox Cluster Configuration:

Elasticsearch Version 0.19.4

2 Master Data Nodes

Both VM servers with 2 CPUs with 8GB memory.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

We are using 0.19.4. I could upgrade to 0.20.5 and try the same thing, but
that would take quite awhile to do since there are 8 nodes in the cluster.

As for the cluster health, I get the following report:

{

  • cluster_name: indextest
  • status: green
  • timed_out: false
  • number_of_nodes: 8
  • number_of_data_nodes: 7
  • active_primary_shards: 134
  • active_shards: 430
  • relocating_shards: 0
  • initializing_shards: 0
  • unassigned_shards: 0

}

I have verified that the 8 nodes IP address in the elasticsearch.yml file
as well:

bootstrap:
mlockall: true
path:
logs: /index/elasticsearch/logs
data: /index/elasticsearch/data
gateway:
type: local
recover_after_nodes: 4
recover_after_time: 5m
expected_nodes: 8
discovery:
zen:
ping_timeout: 3s
minimum_master_nodes: 2
ping:
multicast:
enabled: false
unicast:
enabled: true
hosts: [172.16.47.133
,172.16.47.134
,172.16.47.135
,172.16.47.157
,172.16.46.88
,172.16.47.53
,172.16.47.66
,172.16.47.82]
fd:
ping_interval: 1s
ping_timeout: 30s
ping_retries: 3
script:
default_lang: mvel
cluster:
name: indextest
node:
name: indextest1
master: true
client: false
data: false
action:
auto_create_index: false
disable_delete_all_indices: true
get:
realtime: true
indices:
query:
bool:
max_clause_count : 15000

On Friday, March 1, 2013 6:18:26 AM UTC-6, Clinton Gormley wrote:

On Thu, 2013-02-28 at 19:24 -0800, amos.wood wrote:

Here is my search use case which seems to be failing periodically:
1. Create an index with a 500ms refresh interval
2. Write a document to an index
3. Pause for between 500ms and 5s
4. Attempt to search for a value in 1 field
5. Attempt to search again for the same value in the same field

I'm wondering if your cluster is green and in sync?

Take a look at the cluster health, make sure all your nodes are in
there.

Try doing multiple /_count requests, see if you get back different
results when hitting different nodes.

Perhaps bounce your cluster

clint

In my attempt to search, sometimes I am getting back no results on the
first try with the second search attempt succeeding. Other times, I
am getting back results on both searches. Even more strange,
sometimes I get back a result on the first search but not on the
second search. Does anyone know what may be going on?

I have a gist script which will simulate the issue. I could run this
gist on our production environment cluster and see the issue happen
about every other run. When I run it on our sandbox environment, I
cannot reproduce the issue.

ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub

Production Cluster Configuration:

Elasticsearch Version 0.19.4

1 Master Non Data Node
1 Master Data Node
6 Data Nodes

All VM servers with 4 CPUs with 18GB memory.

Sandbox Cluster Configuration:

Elasticsearch Version 0.19.4

2 Master Data Nodes

Both VM servers with 2 CPUs with 8GB memory.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

After upgrading I didn't see the issue. Also, I then went back to version
0.19.4, and I am not seeing the issue any more either. It may have been a
restart issue or a incomplete configuration issue on the cluster which
resolved when I confirmed all elasticsearch.yml files were correct and
rebooted the cluster. I can say now that the cluster is performing at
great speeds and I am not seeing any latency issues when searching
immediately after the pause of 500ms.

Thanks for your help.

On Friday, March 1, 2013 7:19:41 AM UTC-6, amos.wood wrote:

We are using 0.19.4. I could upgrade to 0.20.5 and try the same thing,
but that would take quite awhile to do since there are 8 nodes in the
cluster.

As for the cluster health, I get the following report:

{

  • cluster_name: indextest
  • status: green
  • timed_out: false
  • number_of_nodes: 8
  • number_of_data_nodes: 7
  • active_primary_shards: 134
  • active_shards: 430
  • relocating_shards: 0
  • initializing_shards: 0
  • unassigned_shards: 0

}

I have verified that the 8 nodes IP address in the elasticsearch.yml file
as well:

bootstrap:
mlockall: true
path:
logs: /index/elasticsearch/logs
data: /index/elasticsearch/data
gateway:
type: local
recover_after_nodes: 4
recover_after_time: 5m
expected_nodes: 8
discovery:
zen:
ping_timeout: 3s
minimum_master_nodes: 2
ping:
multicast:
enabled: false
unicast:
enabled: true
hosts: [172.16.47.133
,172.16.47.134
,172.16.47.135
,172.16.47.157
,172.16.46.88
,172.16.47.53
,172.16.47.66
,172.16.47.82]
fd:
ping_interval: 1s
ping_timeout: 30s
ping_retries: 3
script:
default_lang: mvel
cluster:
name: indextest
node:
name: indextest1
master: true
client: false
data: false
action:
auto_create_index: false
disable_delete_all_indices: true
get:
realtime: true
indices:
query:
bool:
max_clause_count : 15000

On Friday, March 1, 2013 6:18:26 AM UTC-6, Clinton Gormley wrote:

On Thu, 2013-02-28 at 19:24 -0800, amos.wood wrote:

Here is my search use case which seems to be failing periodically:
1. Create an index with a 500ms refresh interval
2. Write a document to an index
3. Pause for between 500ms and 5s
4. Attempt to search for a value in 1 field
5. Attempt to search again for the same value in the same field

I'm wondering if your cluster is green and in sync?

Take a look at the cluster health, make sure all your nodes are in
there.

Try doing multiple /_count requests, see if you get back different
results when hitting different nodes.

Perhaps bounce your cluster

clint

In my attempt to search, sometimes I am getting back no results on the
first try with the second search attempt succeeding. Other times, I
am getting back results on both searches. Even more strange,
sometimes I get back a result on the first search but not on the
second search. Does anyone know what may be going on?

I have a gist script which will simulate the issue. I could run this
gist on our production environment cluster and see the issue happen
about every other run. When I run it on our sandbox environment, I
cannot reproduce the issue.

ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub

Production Cluster Configuration:

Elasticsearch Version 0.19.4

1 Master Non Data Node
1 Master Data Node
6 Data Nodes

All VM servers with 4 CPUs with 18GB memory.

Sandbox Cluster Configuration:

Elasticsearch Version 0.19.4

2 Master Data Nodes

Both VM servers with 2 CPUs with 8GB memory.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

8

thanks for reporting back!

On Friday, March 1, 2013 4:15:02 PM UTC+1, amos.wood wrote:

After upgrading I didn't see the issue. Also, I then went back to version
0.19.4, and I am not seeing the issue any more either. It may have been a
restart issue or a incomplete configuration issue on the cluster which
resolved when I confirmed all elasticsearch.yml files were correct and
rebooted the cluster. I can say now that the cluster is performing at
great speeds and I am not seeing any latency issues when searching
immediately after the pause of 500ms.

Thanks for your help.

On Friday, March 1, 2013 7:19:41 AM UTC-6, amos.wood wrote:

We are using 0.19.4. I could upgrade to 0.20.5 and try the same thing,
but that would take quite awhile to do since there are 8 nodes in the
cluster.

As for the cluster health, I get the following report:

{

  • cluster_name: indextest
  • status: green
  • timed_out: false
  • number_of_nodes: 8
  • number_of_data_nodes: 7
  • active_primary_shards: 134
  • active_shards: 430
  • relocating_shards: 0
  • initializing_shards: 0
  • unassigned_shards: 0

}

I have verified that the 8 nodes IP address in the elasticsearch.yml file
as well:

bootstrap:
mlockall: true
path:
logs: /index/elasticsearch/logs
data: /index/elasticsearch/data
gateway:
type: local
recover_after_nodes: 4
recover_after_time: 5m
expected_nodes: 8
discovery:
zen:
ping_timeout: 3s
minimum_master_nodes: 2
ping:
multicast:
enabled: false
unicast:
enabled: true
hosts: [172.16.47.133
,172.16.47.134
,172.16.47.135
,172.16.47.157
,172.16.46.88
,172.16.47.53
,172.16.47.66
,172.16.47.82]
fd:
ping_interval: 1s
ping_timeout: 30s
ping_retries: 3
script:
default_lang: mvel
cluster:
name: indextest
node:
name: indextest1
master: true
client: false
data: false
action:
auto_create_index: false
disable_delete_all_indices: true
get:
realtime: true
indices:
query:
bool:
max_clause_count : 15000

On Friday, March 1, 2013 6:18:26 AM UTC-6, Clinton Gormley wrote:

On Thu, 2013-02-28 at 19:24 -0800, amos.wood wrote:

Here is my search use case which seems to be failing periodically:
1. Create an index with a 500ms refresh interval
2. Write a document to an index
3. Pause for between 500ms and 5s
4. Attempt to search for a value in 1 field
5. Attempt to search again for the same value in the same field

I'm wondering if your cluster is green and in sync?

Take a look at the cluster health, make sure all your nodes are in
there.

Try doing multiple /_count requests, see if you get back different
results when hitting different nodes.

Perhaps bounce your cluster

clint

In my attempt to search, sometimes I am getting back no results on the
first try with the second search attempt succeeding. Other times, I
am getting back results on both searches. Even more strange,
sometimes I get back a result on the first search but not on the
second search. Does anyone know what may be going on?

I have a gist script which will simulate the issue. I could run this
gist on our production environment cluster and see the issue happen
about every other run. When I run it on our sandbox environment, I
cannot reproduce the issue.

ElasticSearch Script to mimic Real Time Search. However, periodically it doesn't return any search results after waiting even an extra amount of time. · GitHub

Production Cluster Configuration:

Elasticsearch Version 0.19.4

1 Master Non Data Node
1 Master Data Node
6 Data Nodes

All VM servers with 4 CPUs with 18GB memory.

Sandbox Cluster Configuration:

Elasticsearch Version 0.19.4

2 Master Data Nodes

Both VM servers with 2 CPUs with 8GB memory.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.