I have a relatively small index (about 5M docs) and each document has an organization_id field. I need to list all the unique values for this field and I've implemented this using partitions by following this:
It works in most cases.. but I managed to find a few cases where the number of results returned does not sum up to the same amount when changing the number of partitions.
I know my query should return about 19000 values. If I do a single query with only 1 partition or with 5 partitions, I am getting more results then if I do 20 queries with 20 partitions. The missing docs (484 of them) are consistent as they are always the same.
I've double-checked to make sure I was not skipping or missing a partition so I am at loss to explain this behaviour. I am using AWS ElasticSearch 6.2.
any help/comment is welcome