Apm_metrics_dynamic_template does not exist

Hi, I'm trying to setup apm server for tracing purposes. Kibana successfully detects APM server but when I run apm agent i'm getting error like this

Jan 5 15:37:16 elasticapm-server apm-server: 2022-01-05T15:37:16.052+0300#011DEBUG#011[elasticsearch]#011elasticsearch/client.go:523#011Bulk item insert failed (i=0, status=500): {"type":"illegal_state_exception","reason":"Pipeline processor configured for non-existent pipeline [apm_metrics_dynamic_template]"}

APM - Kibana - Elasticsearch 7.6.1

Welcome to our community!

To be able to help you out with this, I've got a few questions:

  • What steps did you perform to get into this state?
  • It seems like some pipelines are missing, did you follow the setup guide for that version?
  • Do you have a mix of versions that have connected to the backing Elasticsearch cluster? This may cause some issues a newer versions of APM Server will set up pipelines that weren't used in 7.6

On another note, it looks like you've opened a PR against the 6.7 branch to add some pipelines that aren't used in that version. Could you confirm that the version that you're using is in fact 7.6.1?

Also, If you're just setting up APM Server why did you chose 7.6.1 and not the latest version 7.16.2?

Hi,
We've followed steps that defined in kibana. And the guide that you gave.
I've mistakenly opened PR aganist 6.7 it should be 7.6 it's my bad sorry.I took wrong definition file . I'm going to fix this mistakes.
Also I've tried both 7.6.1 and 7.6.2 versions result was same for us.

The versions I choose according to Elasticsearch version and it's 7.6.1.
In support matrix shows that APM server 7.0.x-7.16.x***
supports Elastic 7.6.x . But I got another error aganist 7.16.x that's why I choose 7.6.1 APM Server.

I've changed definitions.json according the errors and PR content should be like this.

[{
    "id": "apm",
    "body": {
      "description": "Default enrichment for APM events",
      "processors": [{
          "pipeline": {
            "name": "apm_user_agent"
          }
        },
        {
          "pipeline": {
            "name": "apm_user_geo"
          }
        },
        {
          "pipeline": {
            "name": "apm_ingest_timestamp"
          }
        },
        {
          "pipeline": {
            "name": "apm_remove_span_metadata"
          }
        },
        {
          "pipeline": {
            "name": "apm_metrics_dynamic_template",
            "if": "ctx.processor?.event == 'metric'"
          }
        }
      ]
    }
  },
  {
    "id": "apm_user_agent",
    "body": {
      "description": "Add user agent information for APM events",
      "processors": [{
        "user_agent": {
          "field": "user_agent.original",
          "target_field": "user_agent",
          "ignore_missing": true,
          "ignore_failure": true
        }
      }]
    }
  },
  {
    "id": "apm_remove_span_metadata",
    "body": {
      "description": "Removes metadata fields available already on the parent transaction, to save storage",
      "processors": [{
        "remove": {
          "field": [
            "host",
            "process",
            "user",
            "user_agent",
            "container",
            "kubernetes",
            "service.node",
            "service.version",
            "service.language",
            "service.runtime",
            "service.framework"
          ],
          "if": "ctx.processor?.event == 'span'",
          "ignore_failure": true,
          "ignore_missing": true
        }
      }]
    }
  },
  {
    "id": "apm_ingest_timestamp",
    "body": {
      "description": "Add an ingest timestamp for APM events",
      "processors": [{
        "set": {
          "field": "event.ingested",
          "if": "ctx.processor?.event != 'span'",
          "value": "{{_ingest.timestamp}}"
        }
      }]
    }
  },
  {
    "id": "apm_user_geo",
    "body": {
      "description": "Add user geo information for APM events",
      "processors": [{
        "geoip": {
          "database_file": "GeoLite2-City.mmdb",
          "field": "client.ip",
          "target_field": "client.geo",
          "ignore_missing": true,
          "on_failure": [{
            "remove": {
              "field": "client.ip",
              "ignore_missing": true,
              "ignore_failure": true
            }
          }]
        }
      }]
    }
  },

  {
    "id": "apm_metrics_dynamic_template",
    "body": {
      "description": "Set dynamic_templates for application metrics",
      "processors": [
        {
          "script": {
            "if": "ctx._metric_descriptions != null",
            "source": "Map dynamic_templates = new HashMap();\nfor (entry in ctx._metric_descriptions.entrySet()) {\n  String name = entry.getKey();\n  Map description = entry.getValue();\n  String metric_type = description.type;\n  if (metric_type == \"histogram\") {\n    dynamic_templates[name] = \"histogram\";\n  }\n}\nctx._dynamic_templates = dynamic_templates;\nctx.remove(\"_metric_descriptions\");\n"
          }
        }
      ]
   }
}
]

Thank you for the extra information.

Could you run me through the steps that you took when you installed / configured APM Server?

Did you run the apm-server setup command as specified in the configuring guide? Step 2: Set up and configure | APM Server Reference [7.6] | Elastic. If that command isn't run, then the pipelines and index management won't be created in the backing Elasticsearch cluster.

Hi,
Steps that I've follow:

rpm -i apm-server-7.6.1-x86_64.rpm
set apm-server.host: 0.0.0.0:8200
set kibana.enabled : true
set kibana.host: "[kibana_host]"
set output.elasticsearch.hosts: ["elastic_host"]
set output.elasticsearch.enabled: true

systemctl start apm-server
apm-server setup --index-management
apm-server setup --pipelines

Now I'm getting error like this

Jan 7 16:08:27 elasticapm-server apm-server: 2022-01-07T16:08:27.993+0300#011DEBUG#011[elasticsearch]#011elasticsearch/client.go:523#011Bulk item insert failed (i=0, status=500): {"type":"illegal_state_exception","reason":"Pipeline processor configured for non-existent pipeline [apm_error_grouping_name]"}

when i add this pipeline and feature to definitions.json it's resolves .....

I was unable to reproduce what you're seeing setting up a new Elasticsearch deployment using versions 7.6.1 across the stack.

I've set up the necessary configuration options in the apm-server.yml configuration file:

apm-server:
  host: "localhost:8200"
kibana.enabled : true
kibana.host: ["kibana:5601"]
kibana.username: elastic
kibana.password: dFSVJQecLwpB8UuJvpRvMYZ0
output.elasticsearch.username: elastic
output.elasticsearch.password: dFSVJQecLwpB8UuJvpRvMYZ0
output.elasticsearch.enabled: true
output.elasticsearch.hosts: ["elasticsearch:9200"]

Then ran the following commands:

$ ./apm-server -c apm-server.yml setup --pipelines
Loaded Ingest pipelines
$ ./apm-server -c apm-server.yml setup --index-management
Index setup finished.
$ ./apm-server -c apm-server.yml -e
2022-01-10T10:40:38.223+0800	INFO	[beat]	instance/beat.go:967	Build info	{"system_info": {"build": {"commit": "3b7823bb329e0e5bfe25e106a3b93e8f61d0451f", "libbeat": "7.6.1", "time": "2020-02-28T22:16:31.000Z", "version": "7.6.1"}}}
2022-01-10T10:40:38.224+0800	INFO	instance/beat.go:298	Setup Beat: apm-server; Version: 7.6.1
[...]
2022-01-10T10:40:38.243+0800	INFO	[onboarding]	beater/onboarding.go:35	Publishing onboarding document
2022-01-10T10:40:39.251+0800	INFO	pipeline/output.go:95	Connecting to backoff(elasticsearch(https://elasticsearch:9200))
2022-01-10T10:40:39.798+0800	INFO	elasticsearch/client.go:757	Attempting to connect to Elasticsearch version 7.6.1
2022-01-10T10:40:39.872+0800	INFO	[license]	licenser/es_callback.go:50	Elasticsearch license: Platinum
2022-01-10T10:40:40.305+0800	INFO	[index-management]	idxmgmt/manager.go:84	Overwrite ILM setup is disabled.
2022-01-10T10:40:40.306+0800	INFO	[index-management]	idxmgmt/manager.go:203	Set setup.template.name to 'apm-%{[observer.version]}'.
2022-01-10T10:40:40.306+0800	INFO	[index-management]	idxmgmt/manager.go:205	Set setup.template.pattern to 'apm-%{[observer.version]}*'.
2022-01-10T10:40:40.384+0800	INFO	template/load.go:89	Template apm-7.6.1 already exists and will not be overwritten.
2022-01-10T10:40:40.384+0800	INFO	[index-management]	idxmgmt/manager.go:211	Finished loading index template.
2022-01-10T10:40:40.455+0800	INFO	[index-management.ilm]	ilm/std.go:139	do not generate ilm policy: exists=true, overwrite=false
2022-01-10T10:40:40.456+0800	INFO	[index-management]	idxmgmt/manager.go:240	ILM policy apm-rollover-30-days successfully loaded.
2022-01-10T10:40:40.532+0800	INFO	template/load.go:89	Template apm-7.6.1-transaction already exists and will not be overwritten.
2022-01-10T10:40:40.533+0800	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.6.1-transaction.
2022-01-10T10:40:40.604+0800	INFO	[index-management]	idxmgmt/manager.go:255	Write alias apm-7.6.1-transaction successfully generated.
2022-01-10T10:40:40.678+0800	INFO	template/load.go:89	Template apm-7.6.1-metric already exists and will not be overwritten.
2022-01-10T10:40:40.678+0800	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.6.1-metric.
2022-01-10T10:40:40.749+0800	INFO	[index-management]	idxmgmt/manager.go:255	Write alias apm-7.6.1-metric successfully generated.
2022-01-10T10:40:40.823+0800	INFO	template/load.go:89	Template apm-7.6.1-profile already exists and will not be overwritten.
2022-01-10T10:40:40.823+0800	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.6.1-profile.
2022-01-10T10:40:40.893+0800	INFO	[index-management]	idxmgmt/manager.go:255	Write alias apm-7.6.1-profile successfully generated.
2022-01-10T10:40:40.966+0800	INFO	template/load.go:89	Template apm-7.6.1-error already exists and will not be overwritten.
2022-01-10T10:40:40.966+0800	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.6.1-error.
2022-01-10T10:40:41.040+0800	INFO	[index-management]	idxmgmt/manager.go:255	Write alias apm-7.6.1-error successfully generated.
2022-01-10T10:40:41.178+0800	INFO	template/load.go:89	Template apm-7.6.1-span already exists and will not be overwritten.
2022-01-10T10:40:41.178+0800	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.6.1-span.
2022-01-10T10:40:41.280+0800	INFO	[index-management]	idxmgmt/manager.go:255	Write alias apm-7.6.1-span successfully generated.
2022-01-10T10:40:41.280+0800	INFO	[index-management]	idxmgmt/manager.go:129	Finished index management setup.
2022-01-10T10:40:41.418+0800	INFO	[pipelines]	pipeline/register.go:53	Pipeline already registered: apm
2022-01-10T10:40:41.493+0800	INFO	[pipelines]	pipeline/register.go:53	Pipeline already registered: apm_user_agent
2022-01-10T10:40:41.563+0800	INFO	[pipelines]	pipeline/register.go:53	Pipeline already registered: apm_user_geo
2022-01-10T10:40:41.563+0800	INFO	[pipelines]	pipeline/register.go:56	Registered Ingest Pipelines successfully.
2022-01-10T10:40:41.564+0800	INFO	pipeline/output.go:105	Connection to backoff(elasticsearch(https://elasticsearch:9200)) established
2022-01-10T10:42:28.008+0800	INFO	[request]	middleware/log_middleware.go:97	request accepted	{"request_id": "8fe5dde2-7816-4768-a36b-81397f677632", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "127.0.0.1", "user-agent": "elasticapm-go/1.14.0 go/go1.17.5", "response_code": 202}
[..]

I believe that since you've connected different versions of APM Server to the same Elasticsearch cluster, newer APM Versions may have created index templates that reference pipelines that don't exist.

Could you remove any APM index templates that aren't meant for APM 7.6.1?

You can manage your Index templates through Kibana > Management > Index Management > Index Templates (tab), you may have more than apm-7.6.1- index templates.

Last, you may want to tell APM Server to overwrite any pipelines that are present so they match APM 7.6.1 by setting register.ingest.pipeline.overwrite: true in your configuration file.

Let me know if this helps.