Search query on elasticsearch returning different output every time

Elastic Stack Elasticsearch
1

Hi,

I am facing weird issue on Elasticsearch 6.7.0
Our Elasticsearch cluster is a two-node cluster. which is refreshed daily through automation. The ES nodes are created daily and pushed data to ES from java app.

Some time we observed that the search queries are returning different output if we run in 1 sec interval.

Search query:
POST /event_index_1/_search
{
"size": 100,
"query": {
"terms": {
"attributes.customUniqueID": [
"event-335629.1d80447f.0"
]
}
}
}

The attributes.customUniqueID field is unique for the documents. So this search query should return one document every time.
If the search query runs the first time then it will return 1 document. On the second execution of search query after 1 or 2 second it will return 0 documents. Again on third execution, it returns 1 document as same as first execution.

The output changes every time when we run the search query from 1 document to 0 documents.

The index created with 10 shards and 1 replica.

Does anyone observe the same issue?

Is there any issue with primary and replica shards mismatch??

2

You can use a routing option to ensure you are querying the same shards for testing, see https://www.elastic.co/guide/en/elasticsearch/reference/7.3/search.html#search-routing

this would allow you to figure out, if this is the problem.

Also can you share the _cat/shards/event_index_1 output?

3

Output for _cat/shards/event_index_1
event_index_1 1 r STARTED 2 9.6kb node1_ip I-qrPxk
event_index_1 1 p STARTED 2 9.6kb node2_ip j2TL94D
event_index_1 3 r STARTED 832 1.7mb node1_ip I-qrPxk
event_index_1 3 p STARTED 832 1.6mb node2_ip j2TL94D
event_index_1 9 r STARTED 3 14.5kb node1_ip I-qrPxk
event_index_1 9 p STARTED 2 8kb node2_ip j2TL94D
event_index_1 8 r STARTED 1 6.7kb node1_ip I-qrPxk
event_index_1 8 p STARTED 1 6.7kb node2_ip j2TL94D
event_index_1 4 r STARTED 1 6.6kb node1_ip I-qrPxk
event_index_1 4 p STARTED 0 261b node2_ip j2TL94D
event_index_1 2 r STARTED 2 9.7kb node1_ip I-qrPxk
event_index_1 2 p STARTED 2 9.7kb node2_ip j2TL94D
event_index_1 6 r STARTED 0 261b node1_ip I-qrPxk
event_index_1 6 p STARTED 0 261b node2_ip j2TL94D
event_index_1 5 r STARTED 1 7.3kb node1_ip I-qrPxk
event_index_1 5 p STARTED 1 7.3kb node2_ip j2TL94D
event_index_1 7 r STARTED 3 82.1kb node1_ip I-qrPxk
event_index_1 7 p STARTED 3 82.1kb node2_ip j2TL94D
event_index_1 0 r STARTED 3 81.5kb node1_ip I-qrPxk
event_index_1 0 p STARTED 0 261b node2_ip j2TL94D

Also tried solution provided in below link. Still no luck.

Modified query based on solution given in link
Search query:
POST /event_index_1/_search?routing=event-335629parent
{
"size": 100,
"query": {
"terms": {
"attributes.customUniqueID": [
"event-335629.1d80447f.0"
]
}
}
}

where event-335629parent is the parent id for the docunment and rounting provided while indexing the document.

4

One more observation:

I have stopped both the nodes.
Started both node individually ( nodes not running in same cluster). Executed the search query on both node separately and observed that ES returning the document for only one node where the replica shards are available. Not getting the document on node where the primary shards are presents.

Is it the replica out of sync issue??
if yes the what is the solution for this. And what the reason for replica out of sync issue

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.