Hello Elastic Community,
I am currently facing one persistent performance issue with the search functionality on my website, which uses Elasticsearch to index and retrieve website content. The core problem is that search requests are initially very fast, but after the Elasticsearch index has been running for several days, the response time gradually increases and eventually becomes noticeably slow for visitors. The website itself remains responsive, and other pages that do not perform Elasticsearch queries continue loading normally, but searches that depend on the Elasticsearch index can take several seconds to return results. Restarting the Elasticsearch service temporarily brings the search response time back to normal, but the slowdown gradually returns after the index has been running for an extended period. I am trying to understand what could cause search performance to deteriorate gradually without a major change in website traffic or query structure.
The search system indexes a large amount of content from my website, and users can search for keywords that may appear in titles, descriptions, categories, and other indexed fields. Under normal conditions, a typical search request returns results almost immediately, even when the query matches a large number of documents. However, after extended uptime, similar queries begin taking noticeably longer to complete. The behaviour is not limited to one particular keyword, because both common and relatively specific searches can become slower once the problem develops. I have compared query requests from the beginning of the index lifecycle with requests made several days later, and the query structure is essentially the same. The main difference appears to be the performance of Elasticsearch itself as the index continues running.
I have already checked several Elasticsearch and server metrics while investigating the slowdown. CPU usage does not remain consistently high enough to explain the increased query latency, and the server still has available memory and storage capacity when the issue occurs. I have also monitored the number of documents in the index and understand that the index naturally grows as new website content is added. What I am trying to determine is whether the gradual increase in search latency could be related to index segments, refresh behaviour, deleted documents, shard configuration, or some other aspect of Elasticsearch index maintenance. I have not made significant changes to the query structure during this period, so I am looking for a way to determine whether the performance degradation is caused by the index itself rather than by the website application generating inefficient queries.
I have also reviewed the search requests generated by the website and confirmed that they are relatively simple queries rather than complicated aggregations running across the entire dataset. The application sends the request to Elasticsearch and waits for the response before displaying the search results to the visitor. When Elasticsearch is performing normally, this process is fast and does not noticeably affect the user experience. When the slowdown occurs, the same application code can spend considerably longer waiting for Elasticsearch to return the response. I have added request timing information to my application logs so I can distinguish between time spent preparing the request and time spent waiting for Elasticsearch. The measurements indicate that the majority of the additional delay occurs while waiting for Elasticsearch to complete the search request.
One of the reasons I am finding this difficult to troubleshoot is that restarting Elasticsearch appears to improve the situation temporarily without any changes to the application code or search queries. After the restart, searches become responsive again, but the performance gradually declines as the service continues operating. I do not want to keep relying on service restarts because this is only masking the underlying problem and is not a suitable solution for a production website. I would rather understand what is accumulating or changing inside the Elasticsearch environment over time. I am particularly interested in knowing whether there are specific statistics or diagnostic APIs I should collect before restarting the service so that I can compare the healthy and degraded states and identify what is responsible for the increased latency.
I would appreciate guidance from the Elastic community on how to systematically diagnose this gradual Elasticsearch search-performance degradation. Specifically, I would like recommendations for checking shard and segment statistics, index health, refresh and merge activity, deleted-document counts, JVM behaviour, query latency, and other metrics that could explain why otherwise similar queries become slower after several days. I would also like to know whether there are recommended tools for identifying inefficient queries or determining whether the index structure needs to be changed as it grows. My goal is to find the actual reason search performance deteriorates over time and make the Elasticsearch-powered search functionality on my website remain consistently fast without periodically restarting the service. Sorry for long post!