APM index not refreshing

[philippkahr]

Hello,

I am having troubles with using visualizations for data that does not have any index mapping.

I am not able to refresh the index in Kibana. If I hit the refresh page it just stays at 370 fields. I even deleted the index and readded it again. I have set the following in my apm-server config.

setup.template.fields: "/etc/apm-server/fields.yml"
setup.template.overwrite: true

But still, it does not refresh the index mapping.

I was hoping to build additional visualizations based upon various http fields but I cannot search or do anything with them as long as there is no mapping.

I even deleted the saved objects where apm*. Still no luck.

I tried to readd the mapping via the setup kibana apm dashboard, with apm-server setup, with the refresh button. Everything seems to be doing something but still, no luck.

Everything is 7.3.0 in my setup. I am using java apm with 1.8.0 with Tomcat. APM itself is working fine.

Is there anything that I am missing?

My entire apm config looks like this:

apm-server:
  host: "0.0.0.0:8200"
  rum:
    enabled: true
output.elasticsearch:
  hosts: ["elasticsearch1:9200","elasticsearch2:9200","elasticsearch3:9200"]
  protocol: "https"
  username: "myFancyUser"
  password: "myFancypassword"
  ssl.certificate_authorities: ["/etc/ssl/certs/myCA.pem"]
tags: ["fancy-service"]
setup.template.fields: "/etc/apm-server/fields.yml"
setup.template.overwrite: true

Already tried those:
Refresh index pattern not working telling me that "nested fields are not supported?"
No cached mapping for this field --> APM did no good sadly
Ability to add more exported fields in APM I followed that to the set-tag

Ok, so with https://www.elastic.co/guide/en/apm/agent/js-base/current/transaction-api.html#transaction-add-tags basically using addLabels() I have to define the fields I want to search and index? That kind a fields odd, especially since this targets the RUM and I do not have RUM in use yet.

[Brandon_Kobel]

Hey @philippkahr, have you already tried refreshing your index pattern in Kibana? If you go to Management -> Index Patterns and select the APM index pattern, in the upper right hand of the screen you'll see a "Refresh field list" button, which I've highlighted below:

[philippkahr]

Hi @Brandon_Kobel
Yes, I have done that multiple times as the elastic user in chrome and safari and firefox, but still no luck.

[philippkahr]

I appended the fields.yml to include the two fields important for me and then did a apm-server setup after that I was able to hit the reload index button in Kibana and the two added fields are shown now. However, I cannot search for them properly. I guess I have to do a reindex? If I say http.request.cookies.LASTSKIN exists it tells me that No results match your search criteria however, if I enable http.request.cookies.LASTSKIN as a column I can see that there are values. Even if I hit the + and - sign on the various log lines that include such values it tells me No results match your search criteria. It does not matter if I use http.request.cookies.LASTSKIN OR http.request.cookies.LASTUSERNAME both are not working

My log line looks something like this now:

Since the warning bang is gone, I think it is properly indexed.

I have no idea what I would have todo to be able to filter and visualize that newly created fields.

In my Index Patterns it looks like this:

[philippkahr]

AH! I only applied my new mappings to the wrong apm indices.
What I had to do was this

PUT apm*/_mapping/
{
  "properties": {
      "http.request.cookies.LASTSKIN": {
        "type": "keyword",
        "index": true
      },
      "http.request.cookies.LASTUSERNAME": {
        "type": "keyword",
        "index": true
      }
    }
}

Now it is working for the newly arriving data. So I guess I would have to do a reindex, but that I can do myself.

Thanks for your help!

[Brandon_Kobel]

Thanks for sharing the solution that you figured out here @philippkahr, it's super helpful for others experiencing a similar issue!

[ea1987 (andrea)]

I have the same problem here (no index pattern refresh using the proper button). I will try to update mapping as suggested but I sincerily expected that it would have been done by ES automatically via Kibana button. Is that a bug? The same happened with previous ELK stack version (6.x)
Any other suggestions?
Thanks

[ea1987 (andrea)]

Update: I tried to update mapping using the same API philipp used but with not success

PUT apm*/_mapping/
{
  "properties": {
      "http.request.headers.Content-Length": {
        "type": "long",
        "index": true
      }
    }
}

I got errors (unfortunately I didn't save them). Later, I tried to update apm mapping setting http object dymaic property to true, in this way

PUT /apm*/_mapping
{
  "properties": {
    "http": {
        "dynamic": "true",
        "properties": {
          "request": {
            "properties": {
              "body": {
                "properties": {
                  "bytes": {
                    "type": "long"
                  },
                  "content": {
                    "type": "keyword",
                    "ignore_above": 1024
                  }
                }
              },
              "bytes": {
                "type": "long"
              },
              "headers": {
                "type": "object",
                "enabled": false
              },
              "method": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "referrer": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          },
          "response": {
            "properties": {
              "body": {
                "properties": {
                  "bytes": {
                    "type": "long"
                  },
                  "content": {
                    "type": "keyword",
                    "ignore_above": 1024
                  }
                }
              },
              "bytes": {
                "type": "long"
              },
              "finished": {
                "type": "boolean"
              },
              "headers": {
                "type": "object",
                "enabled": false
              },
              "status_code": {
                "type": "long"
              }
            }
          },
          "version": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      }
  }
}

Once I try to update mapping again inserting http.request.headers.Content-Length field I get the following error:

{
  "error": {
    "root_cause": [
      {
        "type": "mapper_exception",
        "reason": "Can't update attribute for type [http.request.headers.enabled] in index mapping"
      }
    ],
    "type": "mapper_exception",
    "reason": "Can't update attribute for type [http.request.headers.enabled] in index mapping"
  },
  "status": 500
}

Can you please help me on this?
Thanks

***UPDATE
I have tried also in this way but I get the same error

PUT apm*/_mapping/
{
  "properties": {
    "http": {
        "properties": {
          "request": {
            "properties": {
              "headers": {
                "properties": {
                  "Content-Length": {
                    "type": "long"
                  }
                }
              }
            }
          }
        }
    }
  }
}

[ea1987 (andrea)]

I solved it modifying apm fields.yml file, setting new ecs fields and reloading template via proper command:

./apm-server setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'