As I've scaled my cluster and the size of my data, I'm very much wondering how ES handles the search !
I'm still stuck on ES 1.7.2 (difficult to handle migration to a more recent version), on 6 servers with SSD. I perform a search on 2 indices (around 300GB each, splitted on 6 shards / 1 replica). On some searches a big part of documents are matching (like 30-40%).
Unfortunately, results given are not stable, first results are not always the same, most of the time, expected results from the top results are missing. Though when they do happen to pop up, scoring is as expected, ...
Are my shards too big ? Am I missing something about the cluster management ?
2 indices
300GB each, splitted on 6 shards / 1 replica
6 servers
Are my shards too big ?
300 / 6 = 18 gb per shard.
24 shards on 6 nodes = 4 shards per node
That looks correct to me.
results given are not stable
I can't really tell but may be for obscure reasons your primaries and replicas diverged at some point and replicas are inconsistent?
You can use preference=primary IIRC and check that you have consistent results.
If consistent, then may be set the number of replicas to 0 then set it back to 1.
replicas will be copied again from the primaries.
Great ideas David, Merci Well, that does not help much unfortunately.
I tried to reinit replicas anyway. No luck. Issue is that most of the searches are totally ok, but some searches are not and the number of returned results varies from one call to the other.
My only idea around is that, as I return more like 6M of results from 3(M, it gets truncated somewhere in the chain, probably at the shard level. I don't best results get good scores out of a shard and be lost when merging shards results.
Is that possible that a shard won't return all results for some reasons ? (memory issues, incomplete read of the shard because enough results were found, ...)
I'm desesperately trying to upgrade ES but reindexing is a pain, from 1.7.2 to 5.5.0, difficult to handle and migrate all the API calls.
First tries on migrated documents seems to get more consistent. I've also increased the number of shards.
as I return more like 6M of results from 3M, it gets truncated somewhere in the chain, probably at the shard level. I don't best results get good scores out of a shard and be lost when merging shards results.
WDYM? Did you set "size": 6000000? I don't think so but prefer to double-check.
Is that possible that a shard won't return all results for some reasons ? (memory issues, incomplete read of the shard because enough results were found, ...)
Nothing that I can think of. Corrupted shards should be detected so I don't thing that can happen.
But, when you run the query with preference=primary do you have different results?
When this is happening, can you see in the response some failed shards?
Would be great if you could share some response samples.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.