Apm data timestamps wrong => high load and slow responses

Elastic Stack Elasticsearch
1

We have an Elastic Cloud installation running version 9.4.0. Overall, it is quite heavily loaded.

We have multiple dashboards that use APM and RUM data.

We started noticing that the dashboards were becoming slower and slower, and in some cases timing out.

After reviewing and optimizing several things, we finally used Search Profiler and discovered that dashboard panels for the last 15 minutes were accessing multiple indices from past dates, some of them already in the Cold or Frozen tiers.

After checking those indices, we saw that they contained documents mostly within the expected time range, but also some documents with timestamps from dates before the rollover time, and even worse, some documents with timestamps in the future.

In general, these documents seem to correspond to transactions generated from mobile devices, which obviously have an incorrect system date.

Has anyone had this problem before? Any proposals or suggestions?

For now, what we have done is create a workflow that deletes future-dated records using delete_by_query.

However, we still have the problem that many stored indices have very wide timestamp ranges, which do not represent most of the data. Because of this, they are included in searches where they should not be, affecting this and overall response times.

Thanks!

2

Hi @nb71
First, welcome to the community.

Second, yes I have seen similar issues and they can be quite painful as you're noting. This customer had devices that did not have correct system date time similar to you

What is your ingestion architecture for your APM and RUM data? And what is your ingest volume say in event / sec etc (just curious)

Are using elastic legacy, APM or OTEL?

What my other customer did is create an ingest pipeline that compared the event ingested time to the event timestamp.

If the timestamp fell out of a time range, let's say plus or minus 1 hour they set the timestamp to event.ingested or dropped it the event.

I would think about that first like how to keep your data clean.

This is much better than a after the fact delete by query.

I'm also curious on the query side if you have hot Frozen architecture etc. But think about this ingest side first.

Hopefully that helps

3

So you could do something like this.

Pro Tip : Note I would not put this logic in the the @custom pipeline create a sub pipeline and call it much better approach

PUT _ingest/pipeline/normalize_timestamp_from_ingest
{
  "description": "Replace @timestamp with ingest time when variance is greater than 1 hour",
  "processors": [
    {
      "set": {
        "field": "tmp.ingest_time",
        "value": "{{{_ingest.timestamp}}}"
      }
    },
    {
      "set": {
        "field": "@timestamp",
        "value": "{{{tmp.ingest_time}}}",
        "if": "ctx['@timestamp'] == null || Math.abs(java.time.ZonedDateTime.parse(ctx['@timestamp']).toInstant().toEpochMilli() - java.time.ZonedDateTime.parse(ctx.tmp.ingest_time).toInstant().toEpochMilli()) > 3600000"
      }
    },
    {
      "remove": {
        "field": "tmp",
        "ignore_missing": true
      }
    }
  ]
}

Then call that from the traces custom pipeline somethin like this
Make sure it is the correct @custom pipeline

PUT _ingest/pipeline/traces-apm@custom
{
  "description": "Custom pipeline for APM traces",
  "processors": [
    {
      "pipeline": {
        "name": "normalize_timestamp_from_ingest",
        "if": "ctx['@timestamp'] != null"
      }
    }
  ]
}
4

Thank you very much, Stephen.

First, I’ll answer your questions.

This is an Elastic Cloud deployment using all available data tiers: hot, warm, cold, and frozen.

We are using Elastic APM, and we have started migrating some services to OpenTelemetry.

On the ingestion side, we have three Integration/APM Server instances and two ingest nodes, all provided by Elastic Cloud. We are seeing ingestion peaks of up to approximately 10k events per second for APM traces.

Although the correct solution is the ingest pipeline, we initially thought that using delete_by_query would be much more efficient. At that point, we had only seen events with future timestamps, so deleting them once per day seemed much more efficient than adding extra processing to every event.

Unfortunately, we later found that we are also receiving many “old” events with incorrect timestamps, and those also have a negative impact on the number of shards involved in each query.

So it seems there is no other real option than handling this at ingest time.

Thank you for the script suggestion.

From what we have seen, we not only need to add it to the ingest pipeline for APM/APM RUM traces, but also to all the other metrics indices generated by APM.

Regards, and thanks again.

5

Hi @nb71,

Thanks for the update!. I am going to go ahead and share some thoughts and ideas but they are from a far OTOH during my time as a field engineer at Elastic, I’ve helped stabilize and optimize hundreds of clusters, so I’ve seen quite a few setups.

To get a better picture, could you tell me a bit more about your current environment?

  • What CSP/ECH architecture are you on (AWS, Azure, or GCP)?

  • Which profile are you using (CPU, General, etc.)?

Regarding your ingestion setup: Using ingest nodes on ECH is actually an anti-pattern for most use cases, especially observability. It’s often counter-intuitive, but adding them can actually hurt performance. Here’s why:

  1. Traffic Routing: All traffic—not just ingestion—goes through ingest nodes, which adds overhead.

  2. Enrich Processor Inefficiency: Any enrich processors (like GeoIP) become incredibly expensive. Since enrich data must live on data nodes (which ingest nodes can't access), this causes unnecessary network and processing lag.

The Best Practice:

Avoid ingest nodes. If your Hot nodes are CPU-bound, try scaling horizontally or vertically, or switch to a "CPU optimized" profile. I work with clusters handling 10x your volume that run smoothly without ingest nodes, largely by leaning on CPU-optimized profiles to handle the heavy lifting of ingestion, processing, and queries simultaneously.

I suspect with this approach the timestamp fixing processing would be some small fraction of a percent of the overall processing. Note: there is a couple of different ways to write the replace, I showed the simplest.

Regarding Data Tiers:

You mentioned using all tiers (Hot, Warm, Cold, Frozen). Frankly, "Warm" is often the least effective tier for cost-to-performance. We generally don't recommend it for most cases any more. Switching to "Cold" is usually much more efficient, and you can still handle replicas there.

The 2 prevailing O11Y architectures we see now are H/F and H/C/F.

If you refactor your Warm tier, the savings could perhaps cover the costs of upgrading your Hot architecture. We can chat about creating a custom hardware template later, where you can mix and match profiles for each tier.

Other topics such as when and where you're doing the Force merge are important as well... With CPU optimize you can typically do that right on the hot tier. So all searches from then on or more effective... Force merge on hot is also required if you remove the warm tier.

I am 100% sure you can fix this in a cost effective manner (assuming there is nothing completely unusual about your use case)

6

I had this exact issue when working on logs from mobile devices for a streaming application. In our case these logs were uploaded in batches by the client app, and only when that app was running. So "today" the app might upload 100 logs, some from 2-days ago, some from yesterday, some from now/today, but if the device had the wrong date and time, then these 3 sets of logs were all offset by whatever that offset might be. And it was "normal" for logs from N days ago to be uploaded some time later, this was by-design.

By correlating the client logs with the corresponding server-side API logs on a case-by-case basis, you can often determine the clock offset with reasonable precision and, in theory, update the documents (we could, because I wrote some code to do exactly that). In practice though, it was rarely worth the effort, as these logs made up such a small percentage of our overall volume. They skewed some statistics slightly, but were mostly just an annoyance.

I don't think there's a general solution here, since there is no practical way to correct the bad data in flight. So you are choosing between workarounds based on the specific issues it creates in your case.

We already added an ingest_timestamp during ingestion, and that became our workaround. We could exclude logs where abs(@timestamp - ingest_timestamp) exceeded a chosen tolerance from certain queries, dashboards, etc. In our case, these logs accounted for well under 1% of the total volume and were primarily used for aggregate statistics (unique clients per hour/day/week, play sessions, etc.). For those kinds of metrics, the exact number is less important than the overall trend—nobody cares whether there were 27,306 or 27,294 active clients, but they do care if yesterday’s figure was 13,926.

We also considered just deleting them like you mention. There's some merit in a KISS approach, but it's always hard to get approval in corporate envs for throwing any data way.

7

Thank you again, Stephen.

Our deployment is in GCP, using the storage-optimized profile.

I was a bit surprised by your comment about the ingest nodes. I thought that was considered a good practice, so I will need to run some performance tests to see what happens there. The native APM ingest pipelines do use GeoIP, so if that is the case, we would indeed need to remove them.

The data tiers currently in use are something we have wanted to change for some time now. We inherited the setup as it is, and there is definitely some resource waste. In other environments, we also avoid using the warm tier.

In our setup, comparison with the previous week is used heavily, so we need to keep the data available for that period in an efficient way.

I will take your suggestions into account. Thank you!

8

thanks KEvin.

In ouyr case it's strictly APM data, and there are only a dozen transactions from time to time in millions. Enough to polute the datastream, but something qe can erase without problems.

I think may be the apm server sould had an option for this......
Thanks!

9

You would not need to remove them, your hot nodes will act as ingest nodes, everything will work as before, the main difference is that you would not have dedicated ingest nodes.

10

i was refering to remove the ingest nodes. Thanks!

11

Hi @nb71 A little more detail from afar. :slight_smile:

1st you are probably exactly where you should be if you inherited this cluster, as I said I have seen this evolution many times.

Warm Tier (In)Efficiency

The primary reason I am speculating that your Warm tier may be underperforming / not efficient etc is likely due to the Hot tier configuration. If you’re using a Storage Optimized profile for the Hot tier, it might be CPU-constrained, causing the "force merge to 1 segment" step to be skipped. Consequently, these fragmented segments move to the Warm Persistent Disk, which is inefficient and slows down queries significantly.

Furthermore, if you are performing the force merge on the Warm tier before moving data to the Cold tier (which is required by Cold), that resource-intensive operation is competing for CPU cycles on the Warm tier, leading to additional bottlenecks.

Recipe for Success

  • Optimize Profiles: Use a CPU-optimized profile for Ingest, Write, and Query operations.

  • Force Merge on Hot: Perform a "force merge to 1 segment" after the rollover phase on the Hot tier. This makes all subsequent queries significantly faster.

  • Transition Strategy: Move data directly from Hot to Cold. Using fully mounted searchable snapshots on the Cold tier will likely outperform your current Warm tier setup, as it is more efficient for queries since the force merge results in single segments. BTW the HW for Warm / Cold is identical, but because of what I mentioned above Cold is more efficient and I suspect even 1 replica of cold will outperform the previous warm

A Note on Historical Comparison

For historical comparisons, remember to utilize the pre-aggregated APM indices. These are the same indices used in the Curated UI; if you are building custom visualizations, sticking with this pre-aggregated data is the recommended approach for efficient comparisons.

12

or consider going straight from Hot to Frozen, because ...

... whereas Frozen has much faster disks than either. As long as your frozen tier is large enough to avoid cache misses on any performance-sensitive queries, it should do well.

13

So @DavidTurner agree and expanding :slight_smile: (in case anyone else come across this)

  • We are seeing many H/F architectures for O11y Use Cases.
  • We sometime create transforms to pre-agg data if needed for better query performance.
  • And.. What we have been doing is on some of the other providers like AWS we are using SSD on Cold now for specific use cases where it makes sense...
14

Thank you all.
A lot of good insights and tips.