Zlib: invalid header error in node js apm agent

[venkatkumar229 (Venkat Kumar)]

Kibana version: 7.17.3

Elasticsearch version: 7.17.3

APM Server version: 7.17.3

APM Agent language and version: node.js and version 3

Browser version: Chrome

Original install method (e.g. download page, yum, deb, from source, etc.) and version:npm install

Fresh install or upgraded from other version?: Fresh install

Is there anything special in your setup? Directly sending the logs to Elasticsearch cloud apm and nothing in between.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant): {"log.level":"error","@timestamp":"2022-05-25T12:33:20.932Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (400): Unexpected APM Server response\nAPM Server accepted 0 events in the last request\nError: zlib: invalid header"}

Steps to reproduce:
1.
2.
3.

Errors in browser console (if relevant): {"log.level":"error","@timestamp":"2022-05-25T12:33:20.932Z","log":{"logger":"elastic-apm-node"},"ecs":{"version":"1.6.0"},"message":"APM Server transport error (400): Unexpected APM Server response\nAPM Server accepted 0 events in the last request\nError: zlib: invalid header"}

Provide logs and/or server output (if relevant):

[trentm (Trent Mick)]

Hi @venkatkumar229. It sounds like you are attempting to use our Node.js APM agent -- intended for server-side Node.js apps -- in a JS app that runs in the browser. If I understood correctly, you will want to use our APM Real User Monitoring (RUM) JavaScript agent.

Please let me know if I have misunderstood you.

[venkatkumar229 (Venkat Kumar)]

Hi @trentm ,

We are trying to install APM on the server side to get the application metrics. When we try to run apm from local we are able to see the apm service in kibana but when we commit the code to a server and run the application we are not able to see the service and getting the "zlib" invalid header error.

When we are trying from the server we are using a internet gateway inbetween node.js application and Elasticsearch.

Not sure why we are getting the error when we try to run the apm in server side. Could you please help us.

[trentm (Trent Mick)]

@venkatkumar229 When the Node.js APM Agent sends data to the APM Server it uses an HTTP request with gzip compression (and set the content-encoding: gzip header). The "zlib: invalid header" message tells me that APM Server is receiving a request that says the request body is gzipped (i.e. the request headers have content-encoding: gzip), but either the request body is corrupted, or it is no longer compressed gzip data.

we are using a internet gateway inbetween node.js application and Elasticsearch.

Is this some software that you run that is between your Node.js application (where the APM Agent is running) and the APM Server it is configured to talk to (via serverUrl or via the ELASTIC_APM_SERVER_URL environment variable)? If so, is it possible that this gateway does not support gzip'd requests?

[venkatkumar229 (Venkat Kumar)]

Hi @trentm ,

We have set the compression to false in the node.js agent to solve the issue but no luck.

`spanCompressionEnabled: false

[trentm (Trent Mick)]

@venkatkumar229 Sorry, I didn't see your reply until now.

spanCompressionEnabled is about the APM agent taking a number of consecutive spans that are similar and merging them into a single composite span that represents them all (Span compression | APM User Guide [8.2] | Elastic). It is unrelated to the gzip compression that the APM agent uses for its HTTP requests to APM server.

Are you able to give any details about the internet gateway that you are using?