Ingest pipeline for APM traces not working

Elastic Observability APM
,
1

Kibana version:9.0.3

Elasticsearch version:9.0.3

APM Server version:9.0.3

APM Agent language and version: Java, 1.49.0

Browser version: Chrome 140.0.7339.81

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Windows zip file

Fresh install or upgraded from other version? Fresh install

Is there anything special in your setup? Standalone APM server, outputs to Elasticsearch

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant): I’m trying to drop some fields (using ingest pipeline) from the transaction of APM to reduce the index size (traces-apm)

Steps to reproduce:
1.Create component template “traces-apm@custom” :
Settings :
{ "index": { "number_of_replicas": "0", "default_pipeline": "traces-apm-drop-text" } }
2.Create ingest pipeline “traces-apm@custom” ==>

PUT _ingest/pipeline/traces-apm@custom
{
  "description": "Drop .text fields in traces",
  "processors": [
    {
      "remove": {
        "field": [
          "service.name.text",
          "agent.name.text",
          "host.name.text",
          "service.node.name.text"
        ],
        "ignore_missing": true
      }
    }
  ]
  1. push any request to APM server and still observe the “to be dropped” fields :

image
image1048Ă—656 35.5 KB

Errors in browser console (if relevant): none

Provide logs and/or server output (if relevant):none

2

Found something that might be useful, fields that have “dots” in their names means they are nested objects, need dot_expander to work, like this :

PUT _ingest/pipeline/traces-apm@custom

{

  "description": "Drop .text fields in traces",

  "processors": [

    // First, expand the dotted field names into proper nested objects

    {

      "dot_expander": {

        "field": "service.name.text"

      }

    },

    {

      "dot_expander": {

        "field": "agent.name.text"

      }

    },

    {

      "dot_expander": {

        "field": "host.name.text"

      }

    },

    {

      "dot_expander": {

        "field": "service.node.name.text"

      }

    },

    {

      "dot_expander": {

        "field": "process.title.text"

      }

    },

    // Then, remove the now-nested fields

    {

      "remove": {

        "field": [

          "service.name.text",

          "agent.name.text", 

          "host.name.text",

          "service.node.name.text",

          "process.title.text"

        ],

        "ignore_missing": true

      }

    }

  ]

}

But still there is a problem, if i use simulate for the above pipeline it works :
input ==>

PUT _ingest/pipeline/traces-apm@custom

{

  "description": "Drop .text fields in traces",

  "processors": [

    // First, expand the dotted field names into proper nested objects

    {

      "dot_expander": {

        "field": "service.name.text"

      }

    },

    {

      "dot_expander": {

        "field": "agent.name.text"

      }

    },

    {

      "dot_expander": {

        "field": "host.name.text"

      }

    },

    {

      "dot_expander": {

        "field": "service.node.name.text"

      }

    },

    {

      "dot_expander": {

        "field": "process.title.text"

      }

    },

    // Then, remove the now-nested fields

    {

      "remove": {

        "field": [

          "service.name.text",

          "agent.name.text", 

          "host.name.text",

          "service.node.name.text",

          "process.title.text"

        ],

        "ignore_missing": true

      }

    }

  ]

}

Output ==>

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_version": "-3",
        "_id": "_id",
        "_source": {
          "service.node.name": "my-service",
          "agent": {
            "name": {}
          },
          "service.name": "my-service",
          "service": {
            "name": {},
            "node": {
              "name": {}
            }
          }
        },
        "_ingest": {
          "timestamp": "2025-10-05T13:46:33.9303944Z"
        }
      }
    }
  ]
}

But it still doesn’t work in APM traces.
It worked for non-text fields like “service.node.name” and “service.name” but it broke the APM traces tab so I deleted it.
Is there any way to remove these text fields? APM indices are too large.

3

Hi @SamehSaeed

Good research. However, there is a different reason why this is happening.

The .text is defined in the mapping which happens when the document is written which is after the ingest pipeline. So the approach you're taking will never actually work.

If you do not want the .text you need to change the mapping by applying an @custom template and redefine the mapping

So you would define the new mappings here

traces-apm@custom component template

I would caution against this unless you're an expert because that could have other ramifications in the overall app experience

I guess I would go back to. What are you actually trying to solve?

I'm away from keyboard at the moment but I didn't want you to spend too much more tine to solve it with the approach you're trying... It won't work

4

Hello Stephen,

Thanks alot for your reply, this indeed saved much time.

My actual goal is to minimize the size of the index as much as I can. Our management requested a retention of 90 days for apm traces while we can only provide 5 atm. :sweat_smile: Our goal is to shrink down the 70gb daily index to 20-25 but I don't really have a clear vision of how to do it. So i just thought maybe deleting duplicate fields or text fields would help.

5

You are going to be hard-pressed to do that.

You can redefine the mappings and/or drop whole fields ... there should be a number of fields you can drop there are a lot under host etc that you can probably drop.

Also, make sure best_compression is enabled if it is not, you can put that in your custom component template.

1 Like