Remove Specific Fields from APM Data Before Indexing in Elasticsearch

Elastic Observability APM
1

Kibana version: 8.12.2

Elasticsearch version: 8.12.2

APM Server version: 8.12.2

APM Agent language and version: Otel collector-contrib:0.109.0

Browser version: N/A

Original install method (e.g. download page, yum, deb, from source, etc.) and version:

Fresh install or upgraded from other version? Fresh

Is there anything special in your setup? Our primary objective is to remove specific fields from the APM data before it is indexed into Elasticsearch.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

We are working with Elastic APM and aim to exclude certain fields from the data before it's indexed into Elasticsearch. Specifically, I want to remove fields such as agent, agent.name, agent.version, data_stream.dataset,observer.version.​... to reduce metdata storage in elasticsearch.

We are using the following setup:
otel java agent → otel collector → elastic APM → elasticsearch → kibana

2

@Sylvain_Juge Can you please suggest. Thanks!

3

Hi,

Those fields are usually low cardinality and quite short in length, I would be surprised if they would take a significant amount of space, also some of them are used by APM UI, so removing them would very likely make the APM UI unusable.

In your setup, what makes you think the metadata storage is an important element of the storage costs ?

As a first step, you should make sure you have followed the recommendations here to reduce storage requirements.

If you really need to customize the data that is written to Elasticsearch with APM Server, I would suggest to look at APM ingest pipelines

4

@Sylvain_Juge, thank you for your response.

We are just beginning the process of reducing storages. However, as you mentioned, it does not consume significant storage is correct.

I am curious to know whether the recommendations provided here are applicable to the Elastic APM server, given that we use the OpenTelemetry (OTel) agent to send data to Elastic.

As noted in my ticket, our current setup is as follows:
OTel Java Agent → OTel Collector → Elastic APM → Elasticsearch → Kibana

How would these recommendations still be relevant in this context on APM?