Getting `APM Server transport error (503): Unexpected APM Server response` error

prat · December 5, 2020, 1:38pm

Hi Team,

I am seeing APM Server transport error (503): Unexpected APM Server response and
[elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 503 for URL: http://<localhost IP>:7200/intake/v2/events, response code is 503 in /var/log/messages file.

it logs this messages every 2,3 sec.

Dec  5 16:49:46 <HOSTNAME> customa-app-name1: APM Server transport error (503): Unexpected APM Server response
Dec  5 16:49:47 <HOSTNAME> customa-app-name2: APM Server transport error (503): Unexpected APM Server response
Dec  5 16:49:48 <HOSTNAME> customa-app-name3: APM Server transport error (503): Unexpected APM Server response
Dec  5 16:49:53 <HOSTNAME> customa-app-name4: APM Server transport error (503): Unexpected APM Server response
Dec  5 16:49:54 <HOSTNAME> custom-app: APM Server transport error (503): Unexpected APM Server response

config file -

cat /etc/apm-server/apm-server.yml
# Ansible managed

apm-server:
  host: "<localhost IP>:7200"

path.data: /var/lib/apm-server
path.logs: /var/log/apm-server

output.kafka:
  hosts: ["<server1>:3997", "<server2>:3997", "<server3>:3997"]
  topic: "apm"
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  max_message_bytes: 1000000

path.data: /var/lib/apm-server has only meta.json file.

no new logs are generating in /var/log/apm-server/ (last is of Nov 24)

journalctl -fu and /var/log/messages logs.

Dec 05 17:55:31 <APM Server IP> apm-server[125535]: 2020-12-05T17:55:31.390+0530        INFO        [request]        middleware/log_middleware.go:76        request accepted        {"request_id": "eea164df-832a-4fab-90d9-bf21e9fb3607", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "IP", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 202}
Dec 05 17:55:31 <APM Server IP> apm-server[125535]: 2020-12-05T17:55:31.555+0530        INFO        [request]        middleware/log_middleware.go:76        request accepted        {"request_id": "7242bc5f-37c9-44e3-900f-b9cdb8a6f4b5", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "IP", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 202}

Dec 05 17:55:40 <APM Server IP> apm-server[125535]: 2020-12-05T17:55:40.947+0530        ERROR        [request]        middleware/log_middleware.go:74        forbidden request        {"request_id": "96b19cd9-0e8d-48eb-bb7f-18a0a48b0062", "method": "GET", "URL": "/config/v1/agents?service.name=ob-admin-api&service.environment=production", "content_length": 0, "remote_address": "IP", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 403, "error": "forbidden request: endpoint is disabled"}
Dec 05 17:55:40 <APM Server IP> apm-server[125535]: 2020-12-05T17:55:40.956+0530        ERROR        [request]        middleware/log_middleware.go:74        forbidden request        {"request_id": "0d61830d-5a0a-4e7c-a232-17f6fdc18a9a", "method": "GET", "URL": "/config/v1/agents?service.name=ob-admin-api&service.environment=production", "content_length": 0, "remote_address": "IP", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 403, "error": "forbidden request: endpoint is disabled"}
Dec 05 17:55:41 <APM Server IP> apm-server[125535]: 2020-12-05T17:55:41.368+0530        INFO        [request]        middleware/log_middleware.go:76        request accepted        {"request_id": "732d7c5c-de3c-4095-bf81-5080e98edce2", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "IP", "user-agent": "elasticapm-java/1.14.0", "response_code": 202}
Dec 05 17:55:41 <APM Server IP> apm-server[125535]: 2020-12-05T17:55:41.414+0530        INFO        [request]        middleware/log_middleware.go:76        request accepted        {"request_id": "c7913271-7241-4a2f-83a7-2384f257f6b1", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "IP", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 202}

Dec  5 16:49:45 <localhost IP> run.sh: 2020-12-05 16:49:45.154 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 503 for URL: http://<localhost IP>:7200/intake/v2/events, response code is 503
Dec  5 16:49:45 <localhost IP> run.sh: 2020-12-05 16:49:45.154 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - {
Dec  5 16:49:56 <localhost IP> run.sh: 2020-12-05 16:49:56.156 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)
Dec  5 16:50:28 <localhost IP> run.sh: 2020-12-05 16:50:28.969 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 503 for URL: http://<localhost IP>:7200/intake/v2/events, response code is 503
Dec  5 16:50:28 <localhost IP> run.sh: 2020-12-05 16:50:28.969 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - {
Dec  5 16:50:39 <localhost IP> run.sh: 2020-12-05 16:50:39.975 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)

how can I fix this error?

Thanks,

prat · December 5, 2020, 1:56pm

able to telnet to apm-server on port 7200 and also able to self telnet.
apm-server.service is also running.

Eyal_Koren · December 6, 2020, 12:40pm

Please verify that you can curl to the exact same address you use in the server_urls config.

prat · December 7, 2020, 12:43pm

Hi Eyal,

Thanks for your reply.
I am unable find server_urls. Can you please let me know where can I find this.

thanks,

Eyal_Koren · December 7, 2020, 1:00pm

server_urls is how you configure the APM Server/s you want the Java agent to send data to.

prat · December 7, 2020, 1:42pm

Hi @Eyal_Koren,

Thanks for your reply. Can you please let me know where can I look for server_urls.

Thanks,

Eyal_Koren · December 7, 2020, 2:32pm

It is part of the agent setup.
Please post here exactly how you set up the agent to monitor you Java application.

prat · December 9, 2020, 5:25pm

Hi @Eyal_Koren,

Thanks for your reply.

I have java and nodes installed in this system.

for java, I can see below entry in setenv.sh file.

export JAVA_OPTS="-javaagent:/opt/apm/elastic-apm-agent-1.14.0.jar -Delastic.apm.service_name=Tomcat -Delastic.apm.server_url=http://<APM server IP>:7200 -Delastic.apm.enable_log_correlation=true"

-rwxrwxrwx 1 root root 5760548 Oct 15 12:30 /opt/apm/elastic-apm-agent-1.14.0.jar

java version is java version "1.8.0_221"

I can see below error in Catalina.out

2020-12-09 21:04:55.342 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 503 for URL: http://<APM Server IP>:7200/intake/v2/events, response code is 503
2020-12-09 21:05:44.500 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 503 for URL: http://<APM Server IP>:7200/intake/v2/events, response code is 503

I am able to curl to server_url

 curl http://<APM Server IP>:7200
{
  "build_date": "2019-09-27T06:58:27Z",
  "build_sha": "971d864356e4438bf4a799a1fa052cfd0ce680b4",
  "version": "7.4.0"
}
``

for `nodes-apps`, 

I can see below is installed for all `nodes-apps`

# npm list elastic-apm-node
customa-app1@1.0.0 /opt/nodejs-apps/custom-app1
└── elastic-apm-node@3.6.1


it gives below errors for nodes-apps,

Dec 9 21:05:10 custom-app1: APM Server transport error (503): Unexpected APM Server response
Dec 9 21:05:11 custom-app2: APM Server transport error (503): Unexpected APM Server response
Dec 9 21:05:14 custom-app3: APM Server transport error (503): Unexpected APM Server response
Dec 9 21:05:16 custom-app4: APM Server transport error (503): Unexpected APM Server response
Dec 9 21:05:18 custom-app5: APM Server transport error (503): Unexpected APM Server response
Dec 9 21:05:20 run.sh: 2020-12-09 21:05:20.904 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 503 for URL: http://:7200/intake/v2/events, response code is 503


above `run.sh` is present for all nodes-apps in `/opt/nodejs-apps/custom-app1/node_modules/temp/node_modules/rimraf/test/run.sh`

What is causing to produce this issue ?

Thanks,

Eyal_Koren · December 10, 2020, 8:29am

Sorry @prat, my original reply was not very useful, a 503 means that the APM server responded. I was too haste to reply

It seems to be a matter of load, meaning - your agents send more data than your APM server can process, see explanation about 503 and proposed solutions.

The forbidden request errors you see on your server logs are irrelevant, they say your agents try to poll for central configuration (still in beta in 7.4), but it is not enabled.

system · December 31, 2020, 4:29am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Random connectivity error to APM APM nodejs , server	8	694	August 24, 2023
Elastic APM server errors 502 APM nodejs , server	14	1162	September 20, 2023
APM Server transport error: intake response timeout: APM server did not respond within 10s of gzip stream finish APM nodejs , server	5	768	February 1, 2024
APM server did not respond within 10s of gzip stream finish APM docker , server	2	1400	June 13, 2023
Error from NodeJS APM package: "APM Server transport error: error fetching APM Server version" APM nodejs	3	958	March 27, 2023

Getting `APM Server transport error (503): Unexpected APM Server response` error

Related Topics