APM index not refreshing in Kibana - getting Mapper exception

[puja]

Hi,
Some of http fields are not getting refreshed in Kibana for apm-* index pattern due to which I am unable to create visualization and dashboard. I tried refreshing the index pattern in Kibana -Management>Index Pattern and refresh field for apm index but it didnt work.
Also tried adding the field in fields.yml but no luck.

Also, tried PUT/apm*/_mapping like below and got mapper exception -

PUT apm*/_mapping/ { "properties": { "http.response.headers.Access-Control-Allow-Credentials": { "type": "keyword", "index": true }, "http.response.headers.Access-Control-Allow-Origin": { "type": "keyword", "index": true }, "http.response.headers.Cache-Control": { "type": "keyword", "index": true }, "http.response.headers.Content-Length": { "type": "keyword", "index": true }, "http.response.headers.Access-Control-Expose-Headers": { "type": "keyword", "index": true }, "http.response.headers.Content-Security-Policy": { "type": "keyword", "index": true }, "http.response.headers.Connection": { "type": "keyword", "index": true }, "http.response.headers.Content-Type": { "type": "keyword", "index": true }, "http.response.headers.Date": { "type": "keyword", "index": true } } }
After running above lines, got below output -
{ "error" : { "root_cause" : [ { "type" : "mapper_exception", "reason" : "Can't update attribute for type [http.response.headers.enabled] in index mapping" } ], "type" : "mapper_exception", "reason" : "Can't update attribute for type [http.response.headers.enabled] in index mapping" }, "status" : 500 }

[puja]

Looking forward for solution. Can anyone help.

[gil (Gil Raphaelli)]

Hi @puja, welcome!

You're really close. The issue here is that http.response.headers is an object field with enabled: false, and that can't be changed after the index is created. Instead, you want to modify the index template with those changes and have them take effect when the next index is created. Assuming you're on the latest release at this time, 7.8.1, and using the default ILM setup you could take these steps:

1. Add an index template

PUT _template/apm-custom-7.8.1
{
  "index_patterns": [
    "apm-7.8.1*"
  ],
  "mappings": {
    "properties": {
      "http": {
        "properties": {
          "response": {
            "properties": {
              "headers": {
                "properties": {
                  "Content-Length": {
                    "type": "long",
                    "ignore_malformed": true
                  },
                  "Content-Type": {
                    "type": "keyword"
                  },
                }
              }
            }
          }
        }
      }
    }
  }
}

2. Force the index to roll over:

POST apm-7.8.1-transaction/_rollover
{
  "conditions": {
    "max_docs":   "1"
  }
}

3. Submit another transaction

This should happen immediately if your application is running.

4. Update the Kibana index pattern

Click Refresh Field List like before.

Now return to discover and these fields should behave as described:

You can also use apm-server to update the index template instead of doing the PUT request above yourself. Add these to your apm-server.yml and run setup --index-management:

setup.template.overwrite: true
setup.template.append_fields:
- name: "http.response.headers.Content-Type"
  type: keyword

I suggest adding the append_fields to your config so that when you update apm-server, these customizations are retained. More options are described in the docs.

[puja]

Hi @gil,

Thanks for your reply. But I have one more question-

I have around more that 35 http.request.headers and http.response header fields which are not refreshing. So, do I need to add all those around 35 fields manually in apm-server.yml file or there is some other way to append_fields in file.. like using http.request.headers.*

Awaiting for reply..

[gil (Gil Raphaelli)]

It's possible to enable dynamic mappings for headers though I would only recommend this if the number of headers is reasonably bounded. Configure apm-server.yml with:

setup.template.append_fields:
- name: "http.response.headers"
  type: object
  enabled: true
  dynamic: true

I expect all fields will then be mapped as keyword type, including those like Content-Length and Date.

[puja]

After applying above method, is it possible to change the mapping of Content-Length and Date or any field whose mapping is not keyword

[gil (Gil Raphaelli)]

You should be able to combine the approaches to accomplish that.

[puja]

Thanks @gil
This worked for me..