Limit indexed fields

[Emanuel_Velzi]

Hi!

Using Elastic Stack for Application Performance Management (APM) I want to index only certain fields. For example, I do not need all fields related to "observer":

• observer.ephemeral_id
• observer.hostname
• observer.id
  ...

I have modified the fields.yml file, restart the apm-server and my app, but I'm still seeing all the fields indexed by default, just as before.

What am I doing wrong? Somebody has any idea?

Thanks in advance!

[axw (Andrew Wilkins)]

The fields.yml file can be modified to customise the index mappings, but that won't prevent fields from being indexed. To drop fields you can use an ingest node pipeline.

[Emanuel_Velzi]

Is there some way to prevent my java agent from collecting unused data?
I would like to avoid unnecessary network traffic and overhead..

[felixbarny (Felix Barnsteiner)]

All the metadata is sent only once from the agent to the server so dropping metadata doesn't save any overhead.

@Emanuel_Velzi

I have modified the fields.yml file, restart the apm-server and my app, but I'm still seeing all the fields indexed by default, just as before.

Are you trying to avoid storing the fields or do you just don't want an index on those fields? If it's the latter, note that changes in field.yml only change the index template. It's not possible to change the mapping of an existing index. You'll either have to reindex, delete the index, or wait for the index to be rolled over so that a new index with the new mapping from the changed template is created.

There's currently no way to drop specific fields in the Java agent.

[Emanuel_Velzi]

@felixbarny

All the metadata is sent only once from the agent to the server

In my apm-7.6.2-transaction index, I see a lot of data related to observer and host and so on, in each document. This data is sent only once or do you mean something else?

@felixbarny

Are you trying to avoid storing the fields

Yes, I do not need the most of fields in my transaction index, I don't need the data itself.. So, I would like to take only a little document per transaction.

Is this possible?

[felixbarny (Felix Barnsteiner)]

@Emanuel_Velzi

This data is sent only once or do you mean something else?

It's sent only once from APM Agent to APM Server. However, APM Server sends to Elasticsearch in each document.

@Emanuel_Velzi

Yes, I do not need the most of fields in my transaction index, I don't need the data itself.. So, I would like to take only a little document per transaction.

Is this possible?

As Andrew mentioned, the best way would be to use an ingest node to drop the fields.

[Emanuel_Velzi]

Ok thank you both for your help!