Alternating hits totals for subsequent requests

Elastic Stack Elasticsearch
1

Why would I get a different alternating hits total for subsequent match_all
search requests (i.e. curl http://localhost:9200/_search) with zero failed?

i.e. The first time I get 8503251, the next time I get 8479263, the next
time I get 8503251, the next time I get 8479263, and so on, and so on.

My setup is the following:

  • ElasticSearch 0.90.7 with
    • 10 nodes (8 data + 1 primary master + 1 secondary master)
    • 35 indexes w/ 5 shards + 1 replica per index
  • CentOS 6.4 (64 bit)
  • Java 1.7.0_45 (64 bit).

Thoughts?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/93ec8afa-b728-4a6c-a506-288e6259f4ae%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

It shouldn't happen if you are not performing updates in parallel and don't
have documents with a TTL. Maybe you have a plugin that persists data in an
index that is taken into account when summing all hits in all indices?
(would be helpful to know the index which has changing counts).

On Wed, Feb 5, 2014 at 5:27 PM, asanderson a.steven.anderson@gmail.comwrote:

Why would I get a different alternating hits total for subsequent
match_all search requests (i.e. curl http://localhost:9200/_search) with
zero failed?

i.e. The first time I get 8503251, the next time I get 8479263, the next
time I get 8503251, the next time I get 8479263, and so on, and so on.

My setup is the following:

  • Elasticsearch 0.90.7 with
    • 10 nodes (8 data + 1 primary master + 1 secondary master)
    • 35 indexes w/ 5 shards + 1 replica per index
  • CentOS 6.4 (64 bit)
  • Java 1.7.0_45 (64 bit).

Thoughts?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/93ec8afa-b728-4a6c-a506-288e6259f4ae%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j69axbRm31JW-U0-YzMn_KfWYeFbFy%3DHy-e__6ivQi0GA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

We are definitely not performing any updates in parallel nor do we have any
TTLs. Even if we did though, why would the counts alternate on subsequent
requests?

As far as plugins, we are using the analysis-phonetic plugin but
considering this is a match_all query, I wouldn't think it would be a
factor.

How would I determine which index count is different?

On Wednesday, February 5, 2014 11:44:16 AM UTC-5, Adrien Grand wrote:

It shouldn't happen if you are not performing updates in parallel and
don't have documents with a TTL. Maybe you have a plugin that persists data
in an index that is taken into account when summing all hits in all
indices? (would be helpful to know the index which has changing counts).

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/165f691b-03d5-4913-8949-4e45430d6f8f%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Sorry, I thought you said that document counts were changing but it's
actually more than that it's going back and forth to 2 different values
(didn't get that).

This is indeed unexpected. You could try to run

curl http://localhost:9200/_aliases

to make sure all your shards have the same number of documents (they may
however have different numbers of deleted documents, this is not a problem).

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j4jbFksDLVNdBuq393GwF5CttEV-uA_c-tOpjVBOdP70Q%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Actually, I did indeed determine that for a few indexes there were
different doc counts between some shards and their replicas.

Under what circumstances could this happen, and what's the fastest/easiest
way to correct this situation?

On Wednesday, February 5, 2014 12:02:20 PM UTC-5, Adrien Grand wrote:

to make sure all your shards have the same number of documents (they may
however have different numbers of deleted documents, this is not a problem).

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/31799376-685e-47a0-89f8-e066a86802ef%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

On Wed, Feb 5, 2014 at 6:33 PM, asanderson a.steven.anderson@gmail.comwrote:

Actually, I did indeed determine that for a few indexes there were
different doc counts between some shards and their replicas.

Under what circumstances could this happen, and what's the fastest/easiest
way to correct this situation?

This is something that shouldn't happen on a refreshed index. Could you run
a refresh on the indices that have different counts per shard:

curl -XPOST localhost:9200/${index_name}/_refresh

And then show two responses to
curl http://localhost:9200/${index_name}/_search
that return different counts?

Regarding fixing, the easiest way would be to set the number_of_replicas
setting to 0 and then back to its original value: this will deallocate
replica shards and then allocate them again and they will get copied from
the primaries.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j4tE%2BOFmvvD%2BCyCHkQLPz8Rnbc-p%3DSQORwU_0ZPs9Bk2A%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

This is something that shouldn't happen on a refreshed index. Could you
run a refresh on the indices that have different counts per shard:

curl -XPOST localhost:9200/${index_name}/_refresh

And then show two responses to
curl http://localhost:9200/${index_name}/_search
that return different counts?

I did try a refresh via elasticsearch-head but it did not fix the problem
with the replicas.

Regarding fixing, the easiest way would be to set the number_of_replicas
setting to 0 and then back to its original value: this will deallocate
replica shards and then allocate them again and they will get copied from
the primaries.

I actually fixed it with the cluster move requests but this is good to know
too.

Thanks for the responses!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1a8756d8-6029-4055-b5d9-bcabea59f16e%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

8

Take a look at the 'preference' property of the search request body. If you use the user's session id as a custom string, it should prevent the jumping around of values.

Docs: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-preference.html

Jörg recently mentioned this setting in another thread and I think it applies here.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/edc22eb9-6476-46be-b102-c643a62c378a%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

9

Take a look at the 'preference' property of the search request body. If

you use the user's session id as a custom string, it should prevent the
jumping around of values.

Docs:
Elasticsearch Platform — Find real-time answers at scale | Elastic

Ahh...Good to know. Thank you! :slight_smile:

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5408ad00-b74e-466b-a11c-becede51edea%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

10