Unable to add tags-field to APM documents through ingest pipeline

[apt-get_install_skil]

I'm implementing some ingest pipelines to enrich our APM/RUM data. Works fine so far. However, I noticed that when I try to add a tags field, neither Kibana nor Elasticsearch can properly handle it. For example a very basic failure processor:

{
  "append": {
    "field": "tags",
    "value": ["_ingest_pipeline_error"]
  }
}

The "visual" representation of the documents in Kibana's discovery does not show this field. However, clicking on the JSON tab reveals that the tags field is present.

Filtering on the field's existence results in an empty response. I also refreshed the index via the _refresh api multiple times but nothing seems to work. I noticed that there is no explicit mapping for the tags field in the index but this shouldn't be a problem, should it? The same processor works fine when the target field is labels (which has a mapping).

Stack version: 8.6.1 (on-prem)

Kind regards

[kyungeunni (Kyungeun Kim)]

Hi @apt-get_install_skil,

Sorry for getting back to you late!

Dynamic mapping is disabled for traces-apm-* and traces-apm.rum-* so mapping should be configured for a custom field.

I reproduced your case in my local environment without the mapping. Then I updated the component template to have mapping for the tags field for the targeted datastream and I could see that all works fine.

It should work as expected once you add the mapping for tags field in the @custom template. For example, for traces-apm:

PUT _component_template/traces-apm@custom
{
  "template": {
    "mappings": {
      "properties": {
        "tags": {
          "type": "keyword"
        }
      }
    }
  }
}

Also, be sure you rollover to create a new write index for the data stream as the changes do not apply retroactively to existing indices.

Hope this helps.

Kyungeun

[apt-get_install_skil]

Hey Kyungeun,

thanks for your reply!

Yes, makes perfect sense...just didn't expect that dynamic mapping is disabled even for the tags field. In my mind this is a "standard" field just like labels.

But thanks for the clarification.

Regards,
Lukas