We have an Elastic Cloud installation running version 9.4.0. Overall, it is quite heavily loaded.
We have multiple dashboards that use APM and RUM data.
We started noticing that the dashboards were becoming slower and slower, and in some cases timing out.
After reviewing and optimizing several things, we finally used Search Profiler and discovered that dashboard panels for the last 15 minutes were accessing multiple indices from past dates, some of them already in the Cold or Frozen tiers.
After checking those indices, we saw that they contained documents mostly within the expected time range, but also some documents with timestamps from dates before the rollover time, and even worse, some documents with timestamps in the future.
In general, these documents seem to correspond to transactions generated from mobile devices, which obviously have an incorrect system date.
Has anyone had this problem before? Any proposals or suggestions?
For now, what we have done is create a workflow that deletes future-dated records using delete_by_query.
However, we still have the problem that many stored indices have very wide timestamp ranges, which do not represent most of the data. Because of this, they are included in searches where they should not be, affecting this and overall response times.
Thanks!