Limit indexed fields

Elastic Observability APM
,
1

Hi!

Using Elastic Stack for Application Performance Management (APM) I want to index only certain fields. For example, I do not need all fields related to "observer":

  • observer.ephemeral_id
  • observer.hostname
  • observer.id
    ...

I have modified the fields.yml file, restart the apm-server and my app, but I'm still seeing all the fields indexed by default, just as before.

What am I doing wrong? Somebody has any idea?

Thanks in advance!

2

The fields.yml file can be modified to customise the index mappings, but that won't prevent fields from being indexed. To drop fields you can use an ingest node pipeline.

3

Is there some way to prevent my java agent from collecting unused data?
I would like to avoid unnecessary network traffic and overhead..

4

All the metadata is sent only once from the agent to the server so dropping metadata doesn't save any overhead.

Are you trying to avoid storing the fields or do you just don't want an index on those fields? If it's the latter, note that changes in field.yml only change the index template. It's not possible to change the mapping of an existing index. You'll either have to reindex, delete the index, or wait for the index to be rolled over so that a new index with the new mapping from the changed template is created.

There's currently no way to drop specific fields in the Java agent.

5

In my apm-7.6.2-transaction index, I see a lot of data related to observer and host and so on, in each document. This data is sent only once or do you mean something else?

Yes, I do not need the most of fields in my transaction index, I don't need the data itself.. So, I would like to take only a little document per transaction.

Is this possible?

6

It's sent only once from APM Agent to APM Server. However, APM Server sends to Elasticsearch in each document.

As Andrew mentioned, the best way would be to use an ingest node to drop the fields.

7

Ok thank you both for your help!

8

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.