# Getting \`APM Server transport error (503): Unexpected APM Server response\` error

**URL:** <https://discuss.elastic.co/t/getting-apm-server-transport-error-503-unexpected-apm-server-response-error/257733>\
**Category:** APM\
**Tags:** java, nodejs\
**Created:** [December 5, 2020, 1:38pm UTC](https://discuss.elastic.co/t/getting-apm-server-transport-error-503-unexpected-apm-server-response-error/257733 "2020-12-05T13:38:08Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)\
**Post date:** [December 5, 2020, 1:38pm UTC](https://discuss.elastic.co/t/getting-apm-server-transport-error-503-unexpected-apm-server-response-error/257733/1 "2020-12-05T13:38:08Z")

</div>

Hi Team,

I am seeing `APM Server transport error (503): Unexpected APM Server response` and  
`[elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 503 for URL: http://<localhost IP>:7200/intake/v2/events, response code is 503` in `/var/log/messages` file.

it logs this messages every 2,3 sec.

```auto
Dec 5 16:49:46 <HOSTNAME> customa-app-name1: APM Server transport error (503): Unexpected APM Server response
Dec 5 16:49:47 <HOSTNAME> customa-app-name2: APM Server transport error (503): Unexpected APM Server response
Dec 5 16:49:48 <HOSTNAME> customa-app-name3: APM Server transport error (503): Unexpected APM Server response
Dec 5 16:49:53 <HOSTNAME> customa-app-name4: APM Server transport error (503): Unexpected APM Server response
Dec 5 16:49:54 <HOSTNAME> custom-app: APM Server transport error (503): Unexpected APM Server response

```

config file -

```auto
cat /etc/apm-server/apm-server.yml
# Ansible managed

apm-server:
  host: "<localhost IP>:7200"

path.data: /var/lib/apm-server
path.logs: /var/log/apm-server

output.kafka:
  hosts: ["<server1>:3997", "<server2>:3997", "<server3>:3997"]
  topic: "apm"
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  max_message_bytes: 1000000

```

`path.data: /var/lib/apm-server` has only `meta.json` file.

no new logs are generating in `/var/log/apm-server/` (last is of Nov 24)

`journalctl -fu` and `/var/log/messages` logs.

```auto
Dec 05 17:55:31 <APM Server IP> apm-server[125535]: 2020-12-05T17:55:31.390+0530 INFO [request] middleware/log_middleware.go:76 request accepted {"request_id": "eea164df-832a-4fab-90d9-bf21e9fb3607", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "IP", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 202}
Dec 05 17:55:31 <APM Server IP> apm-server[125535]: 2020-12-05T17:55:31.555+0530 INFO [request] middleware/log_middleware.go:76 request accepted {"request_id": "7242bc5f-37c9-44e3-900f-b9cdb8a6f4b5", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "IP", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 202}

Dec 05 17:55:40 <APM Server IP> apm-server[125535]: 2020-12-05T17:55:40.947+0530 ERROR [request] middleware/log_middleware.go:74 forbidden request {"request_id": "96b19cd9-0e8d-48eb-bb7f-18a0a48b0062", "method": "GET", "URL": "/config/v1/agents?service.name=ob-admin-api&service.environment=production", "content_length": 0, "remote_address": "IP", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 403, "error": "forbidden request: endpoint is disabled"}
Dec 05 17:55:40 <APM Server IP> apm-server[125535]: 2020-12-05T17:55:40.956+0530 ERROR [request] middleware/log_middleware.go:74 forbidden request {"request_id": "0d61830d-5a0a-4e7c-a232-17f6fdc18a9a", "method": "GET", "URL": "/config/v1/agents?service.name=ob-admin-api&service.environment=production", "content_length": 0, "remote_address": "IP", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 403, "error": "forbidden request: endpoint is disabled"}
Dec 05 17:55:41 <APM Server IP> apm-server[125535]: 2020-12-05T17:55:41.368+0530 INFO [request] middleware/log_middleware.go:76 request accepted {"request_id": "732d7c5c-de3c-4095-bf81-5080e98edce2", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "IP", "user-agent": "elasticapm-java/1.14.0", "response_code": 202}
Dec 05 17:55:41 <APM Server IP> apm-server[125535]: 2020-12-05T17:55:41.414+0530 INFO [request] middleware/log_middleware.go:76 request accepted {"request_id": "c7913271-7241-4a2f-83a7-2384f257f6b1", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "IP", "user-agent": "elasticapm-node/3.6.1 elastic-apm-http-client/9.4.0 node/8.12.0", "response_code": 202}

Dec 5 16:49:45 <localhost IP> run.sh: 2020-12-05 16:49:45.154 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 503 for URL: http://<localhost IP>:7200/intake/v2/events, response code is 503
Dec 5 16:49:45 <localhost IP> run.sh: 2020-12-05 16:49:45.154 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - {
Dec 5 16:49:56 <localhost IP> run.sh: 2020-12-05 16:49:56.156 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)
Dec 5 16:50:28 <localhost IP> run.sh: 2020-12-05 16:50:28.969 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 503 for URL: http://<localhost IP>:7200/intake/v2/events, response code is 503
Dec 5 16:50:28 <localhost IP> run.sh: 2020-12-05 16:50:28.969 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - {
Dec 5 16:50:39 <localhost IP> run.sh: 2020-12-05 16:50:39.975 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)

```

how can I fix this error?

Thanks,

---

<div class="post-metadata">

**Author:** ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)\
**Post date:** [December 5, 2020, 1:56pm UTC](https://discuss.elastic.co/t/getting-apm-server-transport-error-503-unexpected-apm-server-response-error/257733/2 "2020-12-05T13:56:27Z")

</div>

able to telnet to apm-server on port 7200 and also able to self telnet.  
`apm-server.service` is also running.

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [December 6, 2020, 12:40pm UTC](https://discuss.elastic.co/t/getting-apm-server-transport-error-503-unexpected-apm-server-response-error/257733/3 "2020-12-06T12:40:28Z")

</div>

Please verify that you can `curl` to the exact same address you use in the `server_urls` config.

---

<div class="post-metadata">

**Author:** ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)\
**Post date:** [December 7, 2020, 12:43pm UTC](https://discuss.elastic.co/t/getting-apm-server-transport-error-503-unexpected-apm-server-response-error/257733/4 "2020-12-07T12:43:57Z")

</div>

Hi Eyal,

Thanks for your reply.  
I am unable find `server_urls`. Can you please let me know where can I find this.

thanks,

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [December 7, 2020, 1:00pm UTC](https://discuss.elastic.co/t/getting-apm-server-transport-error-503-unexpected-apm-server-response-error/257733/5 "2020-12-07T13:00:29Z")

</div>

[`server_urls`](https://www.elastic.co/guide/en/apm/agent/java/current/config-reporter.html#config-server-urls) is how you configure the APM Server/s you want the Java agent to send data to.

---

<div class="post-metadata">

**Author:** ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)\
**Post date:** [December 7, 2020, 1:42pm UTC](https://discuss.elastic.co/t/getting-apm-server-transport-error-503-unexpected-apm-server-response-error/257733/6 "2020-12-07T13:42:15Z")

</div>

Hi @Eyal_Koren,

Thanks for your reply. Can you please let me know where can I look for `server_urls`.

Thanks,

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [December 7, 2020, 2:32pm UTC](https://discuss.elastic.co/t/getting-apm-server-transport-error-503-unexpected-apm-server-response-error/257733/7 "2020-12-07T14:32:06Z")

</div>

It is part of the [agent setup](https://www.elastic.co/guide/en/apm/agent/java/current/setup-javaagent.html).  
Please post here exactly how you set up the agent to monitor you Java application.

---

<div class="post-metadata">

**Author:** ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)\
**Post date:** [December 9, 2020, 5:25pm UTC](https://discuss.elastic.co/t/getting-apm-server-transport-error-503-unexpected-apm-server-response-error/257733/8 "2020-12-09T17:25:05Z")

</div>

Hi @Eyal_Koren,

Thanks for your reply.

I have java and nodes installed in this system.

for java, I can see below entry in `setenv.sh` file.

```auto
export JAVA_OPTS="-javaagent:/opt/apm/elastic-apm-agent-1.14.0.jar -Delastic.apm.service_name=Tomcat -Delastic.apm.server_url=http://<APM server IP>:7200 -Delastic.apm.enable_log_correlation=true"

```

```auto
-rwxrwxrwx 1 root root 5760548 Oct 15 12:30 /opt/apm/elastic-apm-agent-1.14.0.jar

```

java version is `java version "1.8.0_221"`

I can see below error in `Catalina.out`

```auto
2020-12-09 21:04:55.342 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 503 for URL: http://<APM Server IP>:7200/intake/v2/events, response code is 503
2020-12-09 21:05:44.500 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 503 for URL: http://<APM Server IP>:7200/intake/v2/events, response code is 503

```

I am able to curl to `server_url`

```auto
 curl http://<APM Server IP>:7200
{
  "build_date": "2019-09-27T06:58:27Z",
  "build_sha": "971d864356e4438bf4a799a1fa052cfd0ce680b4",
  "version": "7.4.0"
}
``

for `nodes-apps`, 

I can see below is installed for all `nodes-apps`

```

`# npm list elastic-apm-node`  
customa-app1@1.0.0 /opt/nodejs-apps/custom-app1  
└── elastic-apm-node@3.6.1

```auto

it gives below errors for nodes-apps, 

```

Dec 9 21:05:10 custom-app1: APM Server transport error (503): Unexpected APM Server response  
Dec 9 21:05:11 custom-app2: APM Server transport error (503): Unexpected APM Server response  
Dec 9 21:05:14 custom-app3: APM Server transport error (503): Unexpected APM Server response  
Dec 9 21:05:16 custom-app4: APM Server transport error (503): Unexpected APM Server response  
Dec 9 21:05:18 custom-app5: APM Server transport error (503): Unexpected APM Server response  
Dec 9 21:05:20 run.sh: 2020-12-09 21:05:20.904 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 503 for URL: http://:7200/intake/v2/events, response code is 503

```auto

above `run.sh` is present for all nodes-apps in `/opt/nodejs-apps/custom-app1/node_modules/temp/node_modules/rimraf/test/run.sh`

What is causing to produce this issue ?

Thanks,
```

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [December 10, 2020, 8:29am UTC](https://discuss.elastic.co/t/getting-apm-server-transport-error-503-unexpected-apm-server-response-error/257733/9 "2020-12-10T08:29:26Z")

</div>

Sorry @prat, my original reply was not very useful, a `503` means that the APM server responded. I was too haste to reply 🙏

It seems to be a matter of load, meaning - your agents send more data than your APM server can process, see [explanation about 503 and proposed solutions](https://www.elastic.co/guide/en/apm/server/7.4/common-problems.html#common-problems).

The `forbidden request` errors you see on your server logs are irrelevant, they say your agents try to poll for [central configuration](https://www.elastic.co/guide/en/kibana/7.4/agent-configuration.html#agent-configuration) (still in beta in 7.4), but it is not enabled.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 31, 2020, 4:29am UTC](https://discuss.elastic.co/t/getting-apm-server-transport-error-503-unexpected-apm-server-response-error/257733/10 "2020-12-31T04:29:35Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.