Failed to load resource: the server responded with a status of 500 (Internal Server Error)

Elastic Observability APM
8

@Agzem

I reproduced and fixed the error. It is a problem in the apm-server.yml file. First, you have to make sure setup.template.name , setup.template.pattern and output.elasticsearch.index/indices configs are set up correctly, the config must match between these three options.

In your config file you set up setup.template.pattern:apm-*, and you change the default index to %{[service.environment]}-apm-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd} when apm-server creates the mappings it looks for the value set on setup.template.pattern, as they don't match, the mappings are created with the default multi-fileds.

"service" : {
          "properties" : {
            "environment" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
...

That's why you got this error:

Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [service.environment]

Here is how you can fix the apm-server.yml:

setup.template.name: "apm-server"
setup.template.pattern: "*-apm-*" # here was the error, your pattern was set as apm-*
output.elasticsearch:
index: "apm-%{[observer.version]}-%{+yyyy.MM.dd}"
  indices:
    - index: "%{[service.environment]}-apm-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}"

You shouldn't specify output.elasticsearch.index with variable parts, if any docs are sent that do not have these variable part set (e.g. service.environment) these docs can’t be ingested and fill up the APM Server queue. That's why I kept output.elasticsearch.index as it is, and added my own index on output.elasticsearch.indices.index.

After the change here is how the mapping looks like:

"service" : {
          "dynamic" : "false",
          "properties" : {
            "environment" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },

Beware that to make it work you must delete the template and indices already created. Be careful you will lose your data if you delete it.

=================================================================================

There's another solution, very similar, but you wouldn't have to be worried about matching all config properties.

setup.template.name: "apm-server"
setup.template.pattern: "apm-*" # Keep the default value here
output.elasticsearch:
index: "apm-%{[observer.version]}-%{+yyyy.MM.dd}"
  indices:
    # place the service.environment after the apm-
    - index: "apm-%{[service.environment]}-%{[observer.version]}-%{[processor.event]}-%{+yyyy.MM.dd}"

With this setup, everything will keep working as it should and you'll have your indices created by each environment you have. And in the APM UI, you don't have to change the indices as well.

Please let me know if you have any further questions.

2 Likes