# How to l limit/sample data stored in metrics-apm.internal?
**URL:** https://discuss.elastic.co/t/how-to-l-limit-sample-data-stored-in-metrics-apm-internal/338760
**Category:** APM
**Tags:** server
**Created:** [July 19, 2023, 9:29am UTC](https://discuss.elastic.co/t/how-to-l-limit-sample-data-stored-in-metrics-apm-internal/338760 "2023-07-19T09:29:12Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Wojciech\_Kwiecien](https://discuss.elastic.co/u/Wojciech_Kwiecien)
#### Post date: [July 19, 2023, 9:29am UTC](https://discuss.elastic.co/t/how-to-l-limit-sample-data-stored-in-metrics-apm-internal/338760/1 "2023-07-19T09:29:12Z")
Hello 🙂
**Kibana version** : 7.17.9
**Elasticsearch version** : 7.17.9
**APM Server version** : 7.17.9
**APM Agent language and version** : dotnet
**Fresh install or upgraded from other version?** upgraded from 7.17.1 to 7.17.9
**Is there anything special in your setup?** We use Kafka, Logstash, we have our index patterns and custom templates
**Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**: I have a problem with a very big amount of data logged in .ds-metrics-apm.internal-default on our cluster. We are after a transition from stand-alone apm server into data streams (Elastic Agent - Fleet managed).
After that transition in this index .ds.metrics-apm.internal-default there are tons of logs for agent. name - dotnet
For the last 24 h were **238,228,987** hits
We observed those problems with a lot of data after we change fleet settings to experimental:true. After this, we have a very high CPU usage on our nodes.
> [@How do I modify queue.mem.events property when the APM server is managed through Fleet integration?](https://discuss.elastic.co/t/how-do-i-modify-queue-mem-events-property-when-the-apm-server-is-managed-through-fleet-integration/294791):
>
> Kibana version: 7.16.2 Elasticsearch version: 7.16.2 APM Server version: 7.16.1 APM Agent language and version: .NET Standard 2.0 Browser version: NA Original install method (e.g. download page, yum, deb, from source, etc.) and version: Fleet integration for APM server (agent installed through deb) Steps to reproduce: Faced quite a few "Queue is full" errors when deploying app using APM instrumentation. Found this [article](https://medium.com/squad-engineering/how-to-optimize-elastic-apm-6f7f6d58bed5) suggesting that I should increase queue.mem.events value in a…
That's how we found so much data here .ds-metrics-apm.internal-default.
Do we have any option to sample ([Transaction sampling | APM User Guide [8.11] | Elastic](https://www.elastic.co/guide/en/apm/guide/current/sampling.html)) those data?
Can we disable those data?
Before data streams, we do not observe data like this with such a big amount of number.
---
### Author:  [@simitt](https://discuss.elastic.co/u/simitt)
#### Post date: [July 19, 2023, 12:48pm UTC](https://discuss.elastic.co/t/how-to-l-limit-sample-data-stored-in-metrics-apm-internal/338760/2 "2023-07-19T12:48:31Z")
Sampling is not supported for the internal metrics.
The internal metrics are by default powering the APM UI from `8.0` onwards. What the server does is that it basically aggregates raw events into aggregated metrics for a certain time interval. The interval is set to `1 minute` by default in `7.17`. The aggregated metrics should actually act as kind of a rollup for raw data, allowing to delete raw data earlier without losing historical key metrics, such as TPM.
The question is why is there such a large amount of aggregated metrics events. The metrics are aggregated along several dimensions, and maybe your services are instrumented to set high cardinality values where we do not expect them, e.g. if there is a random ID part of the `service.name`. This would lead to hitting aggregation bucket limits, and ultimately lead to issuing one metrics event per apm event. (This behavior is improved in `8.x`).
Could you take a look at the apm-server logs and see if there are any errors logged?
---
### Author:  [@simitt](https://discuss.elastic.co/u/simitt)
#### Post date: [July 20, 2023, 9:54am UTC](https://discuss.elastic.co/t/how-to-l-limit-sample-data-stored-in-metrics-apm-internal/338760/3 "2023-07-20T09:54:19Z")
After looking at this again, I actually realised that your screenshot shows `span_breakdown` metrics. Let me change my response accordingly:
These breakdown metrics are used for populating the _Time spent by span type_ graph, [see details](https://www.elastic.co/guide/en/apm/guide/7.17/metricset-indices.html#breakdown-metrics-fields). The metrics were introduced before `7.17` and are unrelated to the usage of elastic agent or the experimental output.
In `7.x` the sampling rate is not applied to transaction breakdown metrics, as any transaction documents are generally retained, and sampling is only applied to span data, see [what data is sampled?](https://www.elastic.co/guide/en/apm/guide/7.17/sampling.html#_what_data_is_sampled).
From `8.0` onwards, the sampling logic and which data are retained vs. which are calculated has changed.
In summary, if you decide to remove the data in this index (e.g. via ingest pipeline), the _Time spent by span type_ visualization is not going to work anymore.
Is it an option for you to upgrade to `8.latest`?
---
### Author:  [@Wojciech\_Kwiecien](https://discuss.elastic.co/u/Wojciech_Kwiecien)
#### Post date: [July 20, 2023, 11:43am UTC](https://discuss.elastic.co/t/how-to-l-limit-sample-data-stored-in-metrics-apm-internal/338760/4 "2023-07-20T11:43:14Z")
Thanks for all your answers.
We have this on our plans for this year. But we can switch to 8x just like that.
It is a whole process to check if everything working ok on lower env clusters and after that, we can switch the prod cluster to 8x
We will try with ingest pipeline first and maybe we will use our own "sampling" to get only 10 % of the data and 90 % will be dropped.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 17, 2023, 11:43am UTC](https://discuss.elastic.co/t/how-to-l-limit-sample-data-stored-in-metrics-apm-internal/338760/5 "2023-08-17T11:43:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.