Data not appearing in elasticsearch after being received by APM server

Kibana version:

Elasticsearch version:

APM Server version:

APM Agent language and version:
Java 8 (Elastic APM Agent 1.35 and 1.36)

Browser version:
Firefox 102.0 (64-bit)

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
Elasticsearch: apt repo, elastic-agent: standalone

Fresh install or upgraded from other version?
Elasticsearch, APM Server (within fleet) upgraded all together from 8.5.3 to 8.6.1

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.
I have nginx in front of APM servers

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Steps to reproduce:

  1. After upgrade from 8.5.3 to 8.6.1, data from one specific host (that belongs to 1 group of APM Servers) are no longer visible in Kibana, alert thrown Document count reported no data in the last 1m for xxx

Errors in browser console (if relevant):
Not relevant

Provide logs and/or server output (if relevant):
Nginx logs shows data being received

xx.xx.xx.xx - - [16/Feb/2023:00:16:20 +0100] "POST /config/v1/agents HTTP/1.1" 304 0 "-" "apm-agent-java/1.36.0 (service-name)"
xx.xx.xx.xx - - [16/Feb/2023:00:16:28 +0100] "POST /intake/v2/events HTTP/1.1" 202 0 "-" "apm-agent-java/1.36.0 (service-name)"
xx.xx.xx.xx - - [16/Feb/2023:00:16:41 +0100] "POST /intake/v2/events HTTP/1.1" 202 0 "-" "apm-agent-java/1.36.0 (service-name)"
xx.xx.xx.xx - - [16/Feb/2023:00:16:50 +0100] "POST /config/v1/agents HTTP/1.1" 304 0 "-" "apm-agent-java/1.36.0 (service-name)"
xx.xx.xx.xx - - [16/Feb/2023:00:16:52 +0100] "POST /intake/v2/events HTTP/1.1" 202 0 "-" "apm-agent-java/1.36.0 (service-name)"
xx.xx.xx.xx - - [16/Feb/2023:00:17:02 +0100] "POST /intake/v2/events HTTP/1.1" 202 0 "-" "apm-agent-java/1.36.0 (service-name)"

When looking at the elastic agent logs, data is received by APM server

{"log.level":"info","@timestamp":"2023-02-15T23:16:28.711Z","message":"request accepted","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"ecs.version":"1.6.0","log.origin":{"file.line":61,"":"middleware/log_middleware.go"},"url.original":"/intake/v2/events","source.address":"xx.xx.xx.xx","":"50d42347-fb2e-4c5f-8710-5586a4db205b","http.request.method":"POST","event.duration":33198072,"log.logger":"request","":"apm-server","user_agent.original":"apm-agent-java/1.36.0 (service-name)","http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-02-15T23:16:52.105Z","message":"request accepted","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"url.original":"/intake/v2/events","user_agent.original":"apm-agent-java/1.36.0 (service-name)","log.origin":{"file.line":61,"":"middleware/log_middleware.go"},"http.request.method":"POST","":"7c67c7ea-4295-41d4-9601-3d6c8199a898","event.duration":18733716,"ecs.version":"1.6.0","http.response.status_code":202,"log.logger":"request","":"apm-server","source.address":"xx.xx.xx.xx","ecs.version":"1.6.0"}

However when browsing Kibana, i see no data from that agent. And also, since apm server that agent is sending data to is having specific namespace, when looking at data stream, example " traces-apm-apm_central" (apm_central) being the namespace, i see it was last updated February 15th, 2023 3:51:26 PM (around the time upgrade of fleet agent was running).

What happened and how to troubleshoot this , please?

Ok, now this is weird.

Also found log stating following:

[elastic_agent.apm_server][error] failed to index event (fail_processor_exception): fail_processor_exception: Document produced by APM Server v8.6.1, which is newer than the installed APM integration (v8.5.0-preview-1663692203). The APM integration must be upgraded.

However APM integration is updated as well...

Elastic agent was restarted on all 3 hosts belonging to this policy, still the same...

Temporary fix: Reinstall APM Integration.
However it's only temporary for next 24 hours. After that issue happens again, then i have to reinstall APM Integration again. It's like this for few days now.

Anyone has any idea how to troubleshoot this further?

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.