Custom mapping in apm index

Elastic Observability APM
1

I have a deployment in cloud elastic using for apm:

apm metrics are being pushed from Nodejs(fastify framwork). I need to add a custom property mapping to the transaction index.

I am able to add the mapping using

PUT apm*/_mapping
{
  "properties": {
      "transaction.custom.body.deviceId": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256,
            "index": true
          }
        },
        "index": true
      }
} 
}

Deployment version: v7.10.1

However during rollover the new index wont have this mapping. I understand that I need to update the mapping in the index template.


       {
         "apm-7.10.0-transaction" : {
           "order" : 2,
           "index_patterns" : [
             "apm-7.10.0-transaction*"
           ],
           "settings" : {
             "index" : {
               "lifecycle" : {
                 "name" : "apm-rollover-30-days",
                 "rollover_alias" : "apm-7.10.0-transaction"
               }
             }
           },
           "mappings" : { },
           "aliases" : { }
         }
       }

The mapping here is seen as empty. Could you help me with adding the mapping to template? Its cloud-elastic, so I am confused if it needs to be added in (user-setting-override)apm-server.yml or directly to template.

Thanks in advance.

2

Hi @Smrithin_N_S

Well there are perhaps an is easier ways to do this...

But to first answer your questions the mappings are defined in the index template

GET _template/apm-7.10.0

This defines the mapping for any index that matches the pattern

"index_patterns" : [
  "apm-7.10.0*"
],

In that there are dynamic template and so that you can add fields and a mapping will automatically be generated.

Option 1) So for example if you added the field you are interested under fields.* or labels.* or even container.labels.* a mapping would automatically be created these would be of type keyword

  "dynamic_templates" : [
    {
      "labels" : {
        "path_match" : "labels.*",
        "mapping" : {
          "type" : "keyword"
        },
        "match_mapping_type" : "string"
      }
    },
    {
      "container.labels" : {
        "path_match" : "container.labels.*",
        "mapping" : {
          "type" : "keyword"
        },
        "match_mapping_type" : "string"
      }
    },
    {
      "fields" : {
        "path_match" : "fields.*",
        "mapping" : {
          "type" : "keyword"
        },
        "match_mapping_type" : "string"
      }
    },
    {
      "docker.container.labels" : {
        "path_match" : "docker.container.labels.*",
        "mapping" : {
          "type" : "keyword"
        },
        "match_mapping_type" : "string"
      }
    },

If you could put your fields under any of these your ready to go... Note you may still need to update the index-pattern

Option 2) really if you just add this field to a document and it is not an integer it will be automatically added with the mapping below and all you would do is refresh the index patter an it would be there... always... then the same would happen when a new index is create... this just leverages Elasticsearch automatic mapping creation

    "my-new-field" : {
      "type" : "text",
      "fields" : {
        "keyword" : {
          "type" : "keyword",
          "ignore_above" : 256
        }
      }
    }

Option 3) Now if you are really set on adding your exact field as above you would add it to this index template ... of course when you update to 7.10.1 you will need to add it etc. that is why I would probably use one of the predefined fields.

We have now introduced composable index template which will probably ease this in the future but it looks like the APM index templates have not migrated to that yet.

Hopefully this help a bit

1 Like
3

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.