Hi, I'm trying to setup apm server for tracing purposes. Kibana successfully detects APM server but when I run apm agent i'm getting error like this
Jan 5 15:37:16 elasticapm-server apm-server: 2022-01-05T15:37:16.052+0300#011DEBUG#011[elasticsearch]#011elasticsearch/client.go:523#011Bulk item insert failed (i=0, status=500): {"type":"illegal_state_exception","reason":"Pipeline processor configured for non-existent pipeline [apm_metrics_dynamic_template]"}
APM - Kibana - Elasticsearch 7.6.1
Welcome to our community!
To be able to help you out with this, I've got a few questions:
On another note, it looks like you've opened a PR against the 6.7 branch to add some pipelines that aren't used in that version. Could you confirm that the version that you're using is in fact 7.6.1?
Also, If you're just setting up APM Server why did you chose 7.6.1 and not the latest version 7.16.2?
Hi,
We've followed steps that defined in kibana. And the guide that you gave.
I've mistakenly opened PR aganist 6.7 it should be 7.6 it's my bad sorry.I took wrong definition file . I'm going to fix this mistakes.
Also I've tried both 7.6.1 and 7.6.2 versions result was same for us.
The versions I choose according to Elasticsearch version and it's 7.6.1.
In support matrix shows that APM server 7.0.x-7.16.x***
supports Elastic 7.6.x . But I got another error aganist 7.16.x that's why I choose 7.6.1 APM Server.
I've changed definitions.json according the errors and PR content should be like this.
[{
"id": "apm",
"body": {
"description": "Default enrichment for APM events",
"processors": [{
"pipeline": {
"name": "apm_user_agent"
}
},
{
"pipeline": {
"name": "apm_user_geo"
}
},
{
"pipeline": {
"name": "apm_ingest_timestamp"
}
},
{
"pipeline": {
"name": "apm_remove_span_metadata"
}
},
{
"pipeline": {
"name": "apm_metrics_dynamic_template",
"if": "ctx.processor?.event == 'metric'"
}
}
]
}
},
{
"id": "apm_user_agent",
"body": {
"description": "Add user agent information for APM events",
"processors": [{
"user_agent": {
"field": "user_agent.original",
"target_field": "user_agent",
"ignore_missing": true,
"ignore_failure": true
}
}]
}
},
{
"id": "apm_remove_span_metadata",
"body": {
"description": "Removes metadata fields available already on the parent transaction, to save storage",
"processors": [{
"remove": {
"field": [
"host",
"process",
"user",
"user_agent",
"container",
"kubernetes",
"service.node",
"service.version",
"service.language",
"service.runtime",
"service.framework"
],
"if": "ctx.processor?.event == 'span'",
"ignore_failure": true,
"ignore_missing": true
}
}]
}
},
{
"id": "apm_ingest_timestamp",
"body": {
"description": "Add an ingest timestamp for APM events",
"processors": [{
"set": {
"field": "event.ingested",
"if": "ctx.processor?.event != 'span'",
"value": "{{_ingest.timestamp}}"
}
}]
}
},
{
"id": "apm_user_geo",
"body": {
"description": "Add user geo information for APM events",
"processors": [{
"geoip": {
"database_file": "GeoLite2-City.mmdb",
"field": "client.ip",
"target_field": "client.geo",
"ignore_missing": true,
"on_failure": [{
"remove": {
"field": "client.ip",
"ignore_missing": true,
"ignore_failure": true
}
}]
}
}]
}
},
{
"id": "apm_metrics_dynamic_template",
"body": {
"description": "Set dynamic_templates for application metrics",
"processors": [
{
"script": {
"if": "ctx._metric_descriptions != null",
"source": "Map dynamic_templates = new HashMap();\nfor (entry in ctx._metric_descriptions.entrySet()) {\n String name = entry.getKey();\n Map description = entry.getValue();\n String metric_type = description.type;\n if (metric_type == \"histogram\") {\n dynamic_templates[name] = \"histogram\";\n }\n}\nctx._dynamic_templates = dynamic_templates;\nctx.remove(\"_metric_descriptions\");\n"
}
}
]
}
}
]
Thank you for the extra information.
Could you run me through the steps that you took when you installed / configured APM Server?
Did you run the apm-server setup command as specified in the configuring guide? Step 2: Set up and configure | APM Server Reference [7.6] | Elastic. If that command isn't run, then the pipelines and index management won't be created in the backing Elasticsearch cluster.
Hi,
Steps that I've follow:
rpm -i apm-server-7.6.1-x86_64.rpm
set apm-server.host: 0.0.0.0:8200
set kibana.enabled : true
set kibana.host: "[kibana_host]"
set output.elasticsearch.hosts: ["elastic_host"]
set output.elasticsearch.enabled: true
systemctl start apm-server
apm-server setup --index-management
apm-server setup --pipelines
Now I'm getting error like this
Jan 7 16:08:27 elasticapm-server apm-server: 2022-01-07T16:08:27.993+0300#011DEBUG#011[elasticsearch]#011elasticsearch/client.go:523#011Bulk item insert failed (i=0, status=500): {"type":"illegal_state_exception","reason":"Pipeline processor configured for non-existent pipeline [apm_error_grouping_name]"}
when i add this pipeline and feature to definitions.json it's resolves .....
I was unable to reproduce what you're seeing setting up a new Elasticsearch deployment using versions 7.6.1 across the stack.
I've set up the necessary configuration options in the apm-server.yml configuration file:
apm-server:
host: "localhost:8200"
kibana.enabled : true
kibana.host: ["kibana:5601"]
kibana.username: elastic
kibana.password: dFSVJQecLwpB8UuJvpRvMYZ0
output.elasticsearch.username: elastic
output.elasticsearch.password: dFSVJQecLwpB8UuJvpRvMYZ0
output.elasticsearch.enabled: true
output.elasticsearch.hosts: ["elasticsearch:9200"]
Then ran the following commands:
$ ./apm-server -c apm-server.yml setup --pipelines
Loaded Ingest pipelines
$ ./apm-server -c apm-server.yml setup --index-management
Index setup finished.
$ ./apm-server -c apm-server.yml -e
2022-01-10T10:40:38.223+0800 INFO [beat] instance/beat.go:967 Build info {"system_info": {"build": {"commit": "3b7823bb329e0e5bfe25e106a3b93e8f61d0451f", "libbeat": "7.6.1", "time": "2020-02-28T22:16:31.000Z", "version": "7.6.1"}}}
2022-01-10T10:40:38.224+0800 INFO instance/beat.go:298 Setup Beat: apm-server; Version: 7.6.1
[...]
2022-01-10T10:40:38.243+0800 INFO [onboarding] beater/onboarding.go:35 Publishing onboarding document
2022-01-10T10:40:39.251+0800 INFO pipeline/output.go:95 Connecting to backoff(elasticsearch(https://elasticsearch:9200))
2022-01-10T10:40:39.798+0800 INFO elasticsearch/client.go:757 Attempting to connect to Elasticsearch version 7.6.1
2022-01-10T10:40:39.872+0800 INFO [license] licenser/es_callback.go:50 Elasticsearch license: Platinum
2022-01-10T10:40:40.305+0800 INFO [index-management] idxmgmt/manager.go:84 Overwrite ILM setup is disabled.
2022-01-10T10:40:40.306+0800 INFO [index-management] idxmgmt/manager.go:203 Set setup.template.name to 'apm-%{[observer.version]}'.
2022-01-10T10:40:40.306+0800 INFO [index-management] idxmgmt/manager.go:205 Set setup.template.pattern to 'apm-%{[observer.version]}*'.
2022-01-10T10:40:40.384+0800 INFO template/load.go:89 Template apm-7.6.1 already exists and will not be overwritten.
2022-01-10T10:40:40.384+0800 INFO [index-management] idxmgmt/manager.go:211 Finished loading index template.
2022-01-10T10:40:40.455+0800 INFO [index-management.ilm] ilm/std.go:139 do not generate ilm policy: exists=true, overwrite=false
2022-01-10T10:40:40.456+0800 INFO [index-management] idxmgmt/manager.go:240 ILM policy apm-rollover-30-days successfully loaded.
2022-01-10T10:40:40.532+0800 INFO template/load.go:89 Template apm-7.6.1-transaction already exists and will not be overwritten.
2022-01-10T10:40:40.533+0800 INFO [index-management] idxmgmt/manager.go:223 Finished template setup for apm-7.6.1-transaction.
2022-01-10T10:40:40.604+0800 INFO [index-management] idxmgmt/manager.go:255 Write alias apm-7.6.1-transaction successfully generated.
2022-01-10T10:40:40.678+0800 INFO template/load.go:89 Template apm-7.6.1-metric already exists and will not be overwritten.
2022-01-10T10:40:40.678+0800 INFO [index-management] idxmgmt/manager.go:223 Finished template setup for apm-7.6.1-metric.
2022-01-10T10:40:40.749+0800 INFO [index-management] idxmgmt/manager.go:255 Write alias apm-7.6.1-metric successfully generated.
2022-01-10T10:40:40.823+0800 INFO template/load.go:89 Template apm-7.6.1-profile already exists and will not be overwritten.
2022-01-10T10:40:40.823+0800 INFO [index-management] idxmgmt/manager.go:223 Finished template setup for apm-7.6.1-profile.
2022-01-10T10:40:40.893+0800 INFO [index-management] idxmgmt/manager.go:255 Write alias apm-7.6.1-profile successfully generated.
2022-01-10T10:40:40.966+0800 INFO template/load.go:89 Template apm-7.6.1-error already exists and will not be overwritten.
2022-01-10T10:40:40.966+0800 INFO [index-management] idxmgmt/manager.go:223 Finished template setup for apm-7.6.1-error.
2022-01-10T10:40:41.040+0800 INFO [index-management] idxmgmt/manager.go:255 Write alias apm-7.6.1-error successfully generated.
2022-01-10T10:40:41.178+0800 INFO template/load.go:89 Template apm-7.6.1-span already exists and will not be overwritten.
2022-01-10T10:40:41.178+0800 INFO [index-management] idxmgmt/manager.go:223 Finished template setup for apm-7.6.1-span.
2022-01-10T10:40:41.280+0800 INFO [index-management] idxmgmt/manager.go:255 Write alias apm-7.6.1-span successfully generated.
2022-01-10T10:40:41.280+0800 INFO [index-management] idxmgmt/manager.go:129 Finished index management setup.
2022-01-10T10:40:41.418+0800 INFO [pipelines] pipeline/register.go:53 Pipeline already registered: apm
2022-01-10T10:40:41.493+0800 INFO [pipelines] pipeline/register.go:53 Pipeline already registered: apm_user_agent
2022-01-10T10:40:41.563+0800 INFO [pipelines] pipeline/register.go:53 Pipeline already registered: apm_user_geo
2022-01-10T10:40:41.563+0800 INFO [pipelines] pipeline/register.go:56 Registered Ingest Pipelines successfully.
2022-01-10T10:40:41.564+0800 INFO pipeline/output.go:105 Connection to backoff(elasticsearch(https://elasticsearch:9200)) established
2022-01-10T10:42:28.008+0800 INFO [request] middleware/log_middleware.go:97 request accepted {"request_id": "8fe5dde2-7816-4768-a36b-81397f677632", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "127.0.0.1", "user-agent": "elasticapm-go/1.14.0 go/go1.17.5", "response_code": 202}
[..]
I believe that since you've connected different versions of APM Server to the same Elasticsearch cluster, newer APM Versions may have created index templates that reference pipelines that don't exist.
Could you remove any APM index templates that aren't meant for APM 7.6.1?
You can manage your Index templates through Kibana > Management > Index Management > Index Templates (tab), you may have more than apm-7.6.1- index templates.
Last, you may want to tell APM Server to overwrite any pipelines that are present so they match APM 7.6.1 by setting register.ingest.pipeline.overwrite: true in your configuration file.
Let me know if this helps.
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.
