APM Server transport error: intake response timeout: APM server did not respond within 10s of gzip stream finish

[Kok_How_Teh]

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text

TIP 1: select at least one tag that further categorizes your topic. For example server for APM Server related questions, java for questions regarding the Elastic APM Java agent, or ui for questions about the APM App within Kibana.

TIP 2: Check out the troubleshooting guide first. Not only will it help you to resolve common problems faster but it also explains in more detail which information we need before we can properly help you.

Kibana version: 8.11.3

Elasticsearch version: 8.11.3

APM Server version: 8.11.3

APM Agent language and version: Node.JS 4.3.0

Browser version:

Original install method (e.g. download page, yum, deb, from source, etc.) and version: ECK

Fresh install or upgraded from other version? Upgrade from 8.11.previous

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I am using ECK 8.11.3 and see the following error message from APM. There is no services shown in the APM integration page in Kibana at all.

[myapp-1 myapp] {"log.level":"error","@timestamp":"2023-12-15T07:06:46.066Z","log.logger":"elastic-apm-node","ecs.version":"8.10.0","message":"APM Server transport error: intake response timeout: APM server did not respond within 10s of gzip stream finish"} 

Steps to reproduce:
1.
2.
3.

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

[simitt (Silvia Mitter)]

Hi @Kok_How_Teh ,

do you see any errors in the APM Server logs?
Could you also provide some more details regarding your setup - from which version did you upgrade and are you seeing the error for an apm app that was previously working as expected?

[Kok_How_Teh]

I upgraded from 8.11.0. The error log provided is from the application which has APM agent installed. Saw the following error in APM server:

[my-apm-server-9c7dd86fc-6crpz apm-server] {"log.level":"error","@timestamp":"2023-12-20T13:29:14.393Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "apm-http")","service.name":"apm-server","ecs.version":"1.6.0"}

[simitt (Silvia Mitter)]

@Kok_How_Teh before we start digging into this, can you confirm that the connection between the same apm agent and apm-server was working successfully in 8.11.0 and that there were no changes at all related to your TLS setup?

[Kok_How_Teh]

Yes, confirmed.