APM Server throwing frequent connection failures against Kibana

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text :slight_smile:

TIP 1: select at least one tag that further categorizes your topic. For example server for APM Server related questions, java for questions regarding the Elastic APM Java agent, or ui for questions about the APM App within Kibana.

TIP 2: Check out the troubleshooting guide first. Not only will it help you to resolve common problems faster but it also explains in more detail which information we need before we can properly help you.

Kibana version: 7.10.2

Elasticsearch version: 7.10.2

APM Server version: 7.10.2

APM Agent language and version: elasticapm-java/1.15.0 and elasticapm-java/1.20.0

Browser version: Chrome Version 89.0.4389.114

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
Docker image

Fresh install or upgraded from other version? upgraded from 7.10.0

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

APM Servers fronted with NGINX Ingress Controller on Kubernetes.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I am getting frequent "context canceled" errors in APM Servers when trying to make a request against Kibana for agent-configuration.

I do see some requests successfully being received by Kibana.

Steps to reproduce:
1.
2.
3.

Errors in browser console (if relevant): n/a

Provide logs and/or server output (if relevant):

ERROR   [request]       middleware/log_middleware.go:99 sending request to kibana failed     {"request_id": "c1e065c1-ea1f-482c-9e0f-9c0cd259521a", "method": "POST", "URL": "/config/v1/agents", "content_length": 654, "remote_address": "10.49.2.42", "user-agent": "elasticapm-java/1.15.0", "event.duration": 4989889838, "response_code": 503, "error": "sending request to kibana failed: Post \"https://kibana.<k8s-service>.svc:5601/api/apm/settings/agent-configuration/search\": context canceled"}

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.