Hi All,
I am having an issue with Elasticsearch on my website where search requests sometimes return incomplete results even though the relevant documents are already present in the Elasticsearch index. The website uses Elasticsearch to provide search functionality for visitors, and normally a search query returns the expected set of matching records without any obvious problem. However, there are certain searches where I know that multiple documents should match the entered keyword, but the website only displays a portion of those results. The missing documents are not consistently the same ones, and repeating the same search after some time can sometimes produce a different result set. What makes this confusing is that when I check the indexed data, the documents appear to exist and contain the fields and values that should make them searchable. The issue therefore seems to be somewhere between the search request being sent from the website and the result set being returned by Elasticsearch, rather than the documents simply never being indexed.
The problem is particularly noticeable when the website has a larger amount of content and users perform searches that should match several documents across the index. For example, a keyword may clearly exist in the relevant field of several indexed documents, but the search response returned to the website may contain only some of them, while another request using the same keyword can occasionally return additional matching documents. I am using the Elasticsearch response directly to build the search results displayed on the website, so when Elasticsearch returns an incomplete set, the frontend has no way to show the missing records. I have checked the actual documents in the index and confirmed that the expected values are present, and I have also tried running similar queries directly against Elasticsearch to determine whether the behavior is caused by the website's search interface. The inconsistent nature of the result makes it difficult to determine whether this is related to the query itself, the index state, shard-level searching, or how the application is processing the Elasticsearch response.
I have already looked at the search request and verified that the website is sending the expected keyword and query parameters rather than accidentally changing the user's input before sending it to Elasticsearch. I have also compared responses from searches that work correctly with responses from searches where some expected documents are missing. In the cases where the issue occurs, Elasticsearch still returns a valid response rather than an obvious error, timeout, or failed request, which means the website treats the response as a successful search and renders whatever results it receives. I do not want to simply increase the number of displayed results because the problem is that some documents that should match appear to be absent from the response in the first place. I am also trying to avoid changing the search query randomly because I would like to understand why documents that are definitely indexed and apparently match the search criteria are not consistently being included in the returned results.
One thing I am particularly unsure about is whether Elasticsearch can temporarily produce different or incomplete result sets because of the way the index is distributed across shards or because of the state of recently indexed documents. The website receives regular content updates, so documents can be added or modified over time, although the search problem is not limited to newly created documents. I would like to understand whether refresh behavior, shard-level execution, query rewriting, pagination parameters, result limits, or another Elasticsearch mechanism could cause an otherwise valid search request to omit documents that appear to match. I am not currently looking for a workaround that simply makes the website issue the same query repeatedly; I would rather identify what Elasticsearch is doing with the query and why the returned hits can differ when the underlying indexed documents have not intentionally changed. If there is a recommended way to inspect the query execution or compare the expected matching documents with the documents Elasticsearch actually considers during the search, that would also help me narrow this down.
Another confusing part is that the issue does not appear to affect every search request, which makes me question whether I am misunderstanding something about how Elasticsearch determines which documents are returned when many documents match. The application does not intentionally remove results after receiving the Elasticsearch response, and I have checked the response handling to make sure the website is not accidentally discarding records before displaying them. I would therefore appreciate some guidance on the Elasticsearch side of the process, particularly how I can determine whether the missing documents were never considered matching, were considered but not included because of result-size or pagination behavior, or were affected by the state of the index at the time the request was executed. I am also interested in knowing which Elasticsearch APIs or diagnostic options would be useful for comparing a problematic search with a normal search without having to make major changes to the production website.
Has anyone experienced a situation where Elasticsearch consistently contains the expected documents but an otherwise successful search request sometimes returns only part of the expected matching set? I would especially like to know what I should check first to determine whether this is related to index refresh timing, shard behavior, query structure, pagination, or another search execution detail. If there is a recommended debugging process for tracing why a specific document is not appearing in a particular Elasticsearch response, I would appreciate an example of how to do that. My main goal is to find out why the website can receive an apparently successful Elasticsearch response while some documents that should match the same search are missing, rather than simply hiding the symptom with repeated requests or frontend changes. Thanks!