Kibana version:
8.6.1
Elasticsearch version:
8.6.1
APM Server version:
8.6.1
APM Agent language and version:
Java 8 (Elastic APM Agent 1.35 and 1.36)
Browser version:
Firefox 102.0 (64-bit)
Original install method (e.g. download page, yum, deb, from source, etc.) and version:
Elasticsearch: apt repo, elastic-agent: standalone
Fresh install or upgraded from other version?
Elasticsearch, APM Server (within fleet) upgraded all together from 8.5.3 to 8.6.1
Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.
I have nginx in front of APM servers
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
Steps to reproduce:
- After upgrade from 8.5.3 to 8.6.1, data from one specific host (that belongs to 1 group of APM Servers) are no longer visible in Kibana, alert thrown
Document count reported no data in the last 1m for xxx
Errors in browser console (if relevant):
Not relevant
Provide logs and/or server output (if relevant):
Nginx logs shows data being received
xx.xx.xx.xx - - [16/Feb/2023:00:16:20 +0100] "POST /config/v1/agents HTTP/1.1" 304 0 "-" "apm-agent-java/1.36.0 (service-name)"
xx.xx.xx.xx - - [16/Feb/2023:00:16:28 +0100] "POST /intake/v2/events HTTP/1.1" 202 0 "-" "apm-agent-java/1.36.0 (service-name)"
xx.xx.xx.xx - - [16/Feb/2023:00:16:41 +0100] "POST /intake/v2/events HTTP/1.1" 202 0 "-" "apm-agent-java/1.36.0 (service-name)"
xx.xx.xx.xx - - [16/Feb/2023:00:16:50 +0100] "POST /config/v1/agents HTTP/1.1" 304 0 "-" "apm-agent-java/1.36.0 (service-name)"
xx.xx.xx.xx - - [16/Feb/2023:00:16:52 +0100] "POST /intake/v2/events HTTP/1.1" 202 0 "-" "apm-agent-java/1.36.0 (service-name)"
xx.xx.xx.xx - - [16/Feb/2023:00:17:02 +0100] "POST /intake/v2/events HTTP/1.1" 202 0 "-" "apm-agent-java/1.36.0 (service-name)"
When looking at the elastic agent logs, data is received by APM server
{"log.level":"info","@timestamp":"2023-02-15T23:16:28.711Z","message":"request accepted","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0","log.origin":{"file.line":61,"file.name":"middleware/log_middleware.go"},"url.original":"/intake/v2/events","source.address":"xx.xx.xx.xx","http.request.id":"50d42347-fb2e-4c5f-8710-5586a4db205b","http.request.method":"POST","event.duration":33198072,"log.logger":"request","service.name":"apm-server","user_agent.original":"apm-agent-java/1.36.0 (service-name)","http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-02-15T23:16:52.105Z","message":"request accepted","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"url.original":"/intake/v2/events","user_agent.original":"apm-agent-java/1.36.0 (service-name)","log.origin":{"file.line":61,"file.name":"middleware/log_middleware.go"},"http.request.method":"POST","http.request.id":"7c67c7ea-4295-41d4-9601-3d6c8199a898","event.duration":18733716,"ecs.version":"1.6.0","http.response.status_code":202,"log.logger":"request","service.name":"apm-server","source.address":"xx.xx.xx.xx","ecs.version":"1.6.0"}
However when browsing Kibana, i see no data from that agent. And also, since apm server that agent is sending data to is having specific namespace, when looking at data stream, example " traces-apm-apm_central" (apm_central) being the namespace, i see it was last updated February 15th, 2023 3:51:26 PM (around the time upgrade of fleet agent was running).
What happened and how to troubleshoot this , please?
