Getting `APM Server transport error (503): Unexpected APM Server response` error

Elastic Observability APM
,
1

Hi Team,

I am seeing APM Server transport error (503): Unexpected APM Server response and
[elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 503 for URL: http://<localhost IP>:7200/intake/v2/events, response code is 503 in /var/log/messages file.

it logs this messages every 2,3 sec.

Dec  5 16:49:46 <HOSTNAME> customa-app-name1: APM Server transport error (503): Unexpected APM Server response
Dec  5 16:49:47 <HOSTNAME> customa-app-name2: APM Server transport error (503): Unexpected APM Server response
Dec  5 16:49:48 <HOSTNAME> customa-app-name3: APM Server transport error (503): Unexpected APM Server response
Dec  5 16:49:53 <HOSTNAME> customa-app-name4: APM Server transport error (503): Unexpected APM Server response
Dec  5 16:49:54 <HOSTNAME> custom-app: APM Server transport error (503): Unexpected APM Server response

config file -

cat /etc/apm-server/apm-server.yml
# Ansible managed

apm-server:
  host: "<localhost IP>:7200"

path.data: /var/lib/apm-server
path.logs: /var/log/apm-server

output.kafka:
  hosts: ["<server1>:3997", "<server2>:3997", "<server3>:3997"]
  topic: "apm"
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  max_message_bytes: 1000000

path.data: /var/lib/apm-server has only meta.json file.

no new logs are generating in /var/log/apm-server/ (last is of Nov 24)

journalctl -fu and /var/log/messages logs.

Dec 05 17:55:31 <APM Server IP> apm-server[125535]: 2020-12-05T17:55:31.390+0530        INFO        [request]        middleware/log_middleware.go:76        request accepted        {"request_id": "eea164df-832a-4fab-90d9-bf21e9fb3607", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "IP", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 202}
Dec 05 17:55:31 <APM Server IP> apm-server[125535]: 2020-12-05T17:55:31.555+0530        INFO        [request]        middleware/log_middleware.go:76        request accepted        {"request_id": "7242bc5f-37c9-44e3-900f-b9cdb8a6f4b5", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "IP", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 202}

Dec 05 17:55:40 <APM Server IP> apm-server[125535]: 2020-12-05T17:55:40.947+0530        ERROR        [request]        middleware/log_middleware.go:74        forbidden request        {"request_id": "96b19cd9-0e8d-48eb-bb7f-18a0a48b0062", "method": "GET", "URL": "/config/v1/agents?service.name=ob-admin-api&service.environment=production", "content_length": 0, "remote_address": "IP", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 403, "error": "forbidden request: endpoint is disabled"}
Dec 05 17:55:40 <APM Server IP> apm-server[125535]: 2020-12-05T17:55:40.956+0530        ERROR        [request]        middleware/log_middleware.go:74        forbidden request        {"request_id": "0d61830d-5a0a-4e7c-a232-17f6fdc18a9a", "method": "GET", "URL": "/config/v1/agents?service.name=ob-admin-api&service.environment=production", "content_length": 0, "remote_address": "IP", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 403, "error": "forbidden request: endpoint is disabled"}
Dec 05 17:55:41 <APM Server IP> apm-server[125535]: 2020-12-05T17:55:41.368+0530        INFO        [request]        middleware/log_middleware.go:76        request accepted        {"request_id": "732d7c5c-de3c-4095-bf81-5080e98edce2", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "IP", "user-agent": "elasticapm-java/1.14.0", "response_code": 202}
Dec 05 17:55:41 <APM Server IP> apm-server[125535]: 2020-12-05T17:55:41.414+0530        INFO        [request]        middleware/log_middleware.go:76        request accepted        {"request_id": "c7913271-7241-4a2f-83a7-2384f257f6b1", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "IP", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 202}

Dec  5 16:49:45 <localhost IP> run.sh: 2020-12-05 16:49:45.154 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 503 for URL: http://<localhost IP>:7200/intake/v2/events, response code is 503
Dec  5 16:49:45 <localhost IP> run.sh: 2020-12-05 16:49:45.154 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - {
Dec  5 16:49:56 <localhost IP> run.sh: 2020-12-05 16:49:56.156 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)
Dec  5 16:50:28 <localhost IP> run.sh: 2020-12-05 16:50:28.969 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 503 for URL: http://<localhost IP>:7200/intake/v2/events, response code is 503
Dec  5 16:50:28 <localhost IP> run.sh: 2020-12-05 16:50:28.969 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - {
Dec  5 16:50:39 <localhost IP> run.sh: 2020-12-05 16:50:39.975 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)

how can I fix this error?

Thanks,

2

able to telnet to apm-server on port 7200 and also able to self telnet.
apm-server.service is also running.

3

Please verify that you can curl to the exact same address you use in the server_urls config.

4

Hi Eyal,

Thanks for your reply.
I am unable find server_urls. Can you please let me know where can I find this.

thanks,

5

server_urls is how you configure the APM Server/s you want the Java agent to send data to.

6

Hi @Eyal_Koren,

Thanks for your reply. Can you please let me know where can I look for server_urls.

Thanks,

7

It is part of the agent setup.
Please post here exactly how you set up the agent to monitor you Java application.

8

Hi @Eyal_Koren,

Thanks for your reply.

I have java and nodes installed in this system.

for java, I can see below entry in setenv.sh file.

export JAVA_OPTS="-javaagent:/opt/apm/elastic-apm-agent-1.14.0.jar -Delastic.apm.service_name=Tomcat -Delastic.apm.server_url=http://<APM server IP>:7200 -Delastic.apm.enable_log_correlation=true"

-rwxrwxrwx 1 root root 5760548 Oct 15 12:30 /opt/apm/elastic-apm-agent-1.14.0.jar

java version is java version "1.8.0_221"

I can see below error in Catalina.out

2020-12-09 21:04:55.342 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 503 for URL: http://<APM Server IP>:7200/intake/v2/events, response code is 503
2020-12-09 21:05:44.500 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 503 for URL: http://<APM Server IP>:7200/intake/v2/events, response code is 503

I am able to curl to server_url

 curl http://<APM Server IP>:7200
{
  "build_date": "2019-09-27T06:58:27Z",
  "build_sha": "971d864356e4438bf4a799a1fa052cfd0ce680b4",
  "version": "7.4.0"
}
``

for `nodes-apps`, 

I can see below is installed for all `nodes-apps`

# npm list elastic-apm-node
customa-app1@1.0.0 /opt/nodejs-apps/custom-app1
└── elastic-apm-node@3.6.1


it gives below errors for nodes-apps,

Dec 9 21:05:10 custom-app1: APM Server transport error (503): Unexpected APM Server response
Dec 9 21:05:11 custom-app2: APM Server transport error (503): Unexpected APM Server response
Dec 9 21:05:14 custom-app3: APM Server transport error (503): Unexpected APM Server response
Dec 9 21:05:16 custom-app4: APM Server transport error (503): Unexpected APM Server response
Dec 9 21:05:18 custom-app5: APM Server transport error (503): Unexpected APM Server response
Dec 9 21:05:20 run.sh: 2020-12-09 21:05:20.904 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 503 for URL: http://:7200/intake/v2/events, response code is 503


above `run.sh` is present for all nodes-apps in `/opt/nodejs-apps/custom-app1/node_modules/temp/node_modules/rimraf/test/run.sh`

What is causing to produce this issue ?

Thanks,
9

Sorry @prat, my original reply was not very useful, a 503 means that the APM server responded. I was too haste to reply :pray:

It seems to be a matter of load, meaning - your agents send more data than your APM server can process, see explanation about 503 and proposed solutions.

The forbidden request errors you see on your server logs are irrelevant, they say your agents try to poll for central configuration (still in beta in 7.4), but it is not enabled.

10

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.