Apm agent cannot connect to remote apm server

Kibana version:
7.4.0
Elasticsearch version:
7.4.0
APM Server version:
7.4.0
APM Agent language and version:
Java, 1.11.0

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

my apm-server is on Kubernetes, and has nginx-proxy in front.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

The elastic-apm-agent and the java application reside on my local system, I am able to send logs when I run kubectl port forward apm-server 8200 but I get 502 status when I specify nginx-proxy loadbalancer url in elasticapm.properties file.

Provide logs and/or server output (if relevant):

my elasticapm.properties file looks like this:

service_name=demo
application_packages=ca.com.elastic
server_urls=https://172.20.11.188
verify_server_cert=false
log_level=debug

apm-server.yaml file:

apm-server.yml: |
host: "0.0.0.0:8200"
setup.template.settings:
index:
number_of_shards: 1
codec: best_compression
setup.kibana:
host: "< kibana endpoint >"
output.elasticsearch:
hosts: ["elasticsearch:9200"]

logs from elastic-apm-agent:

2019-11-22 11:25:03.188 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Starting new request to https://172.20.11.188/intake/v2/events
2019-11-22 11:25:03.224 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Scheduling request timeout in 10s
2019-11-22 11:25:13.225 [apm-request-timeout-timer] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Request flush because the request timeout occurred
2019-11-22 11:25:13.235 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving FLUSH event (sequence 1)
2019-11-22 11:25:13.238 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Flushing 1386 uncompressed 601 compressed bytes
2019-11-22 11:25:13.240 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 0 seconds (+/-10%)
2019-11-22 11:25:13.240 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 502 for URL: https://172.20.11.188/intake/v2/events, response code is 502
2019-11-22 11:25:13.240 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Sending payload to APM server failed
java.io.IOException: Server returned HTTP response code: 502 for URL: https://172.20.11.188/intake/v2/events
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.flush(IntakeV2ReportingEventHandler.java:224)
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.handleEvent(IntakeV2ReportingEventHandler.java:128)
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:116)
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:50)
at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
at java.lang.Thread.run(Thread.java:748)
2019-11-22 11:25:13.243 [apm-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler -

502 Bad Gateway

502 Bad Gateway


nginx/1.13.12

Logs from nginx proxy pod:

2019/11/22 19:24:35 [error] 10#10: *309 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.6.1, server: , request: "POST /config/v1/agents HTTP/1.1", upstream: "http://10.19.72.106:8200/config/v1/agents", host: "172.20.11.188"
10.20.6.1 - - [22/Nov/2019:19:24:35 +0000] "POST /config/v1/agents HTTP/1.1" 502 174 "-" "elasticapm-java/1.11.0" "-"
2019/11/22 19:24:44 [info] 10#10: *309 client 10.20.6.1 closed keepalive connection
2019/11/22 19:25:43 [error] 10#10: *311 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.6.1, server: , request: "POST /intake/v2/events HTTP/1.1", upstream: "http://10.19.72.106:8200/intake/v2/events", host: "172.20.11.188"
10.20.6.1 - - [22/Nov/2019:19:25:43 +0000] "POST /intake/v2/events HTTP/1.1" 502 174 "-" "elasticapm-java/1.11.0" "-"
2019/11/22 19:25:49 [info] 10#10: *311 client 10.20.6.1 closed keepalive connection
2019/11/22 19:26:13 [error] 10#10: *313 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.6.1, server: , request: "POST /intake/v2/events HTTP/1.1", upstream: "http://10.19.72.106:8200/intake/v2/events", host: "172.20.11.188"
10.20.6.1 - - [22/Nov/2019:19:26:13 +0000] "POST /intake/v2/events HTTP/1.1" 502 174 "-" "elasticapm-java/1.11.0" "-"
2019/11/22 19:26:26 [info] 10#10: *313 client 10.20.6.1 closed keepalive connection
10.20.6.1 - - [22/Nov/2019:19:27:13 +0000] "POST /intake/v2/events HTTP/1.1" 502 174 "-" "elasticapm-java/1.11.0" "-"
2019/11/22 19:27:13 [error] 10#10: *315 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.6.1, server: , request: "POST /intake/v2/events HTTP/1.1", upstream: "http://10.19.72.106:8200/intake/v2/events", host: "172.20.11.188"
2019/11/22 19:27:33 [info] 10#10: *315 client 10.20.6.1 closed keepalive connection
2019/11/22 19:27:43 [error] 10#10: *317 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.6.1, server: , request: "POST /intake/v2/events HTTP/1.1", upstream: "http://10.19.72.106:8200/intake/v2/events", host: "172.20.11.188"

Any ideas?

Please start by verifying your NginX configurations and see that the APM server is properly defined and reachable.
Is 10.19.72.106 indeed the correct APM server IP? Is it reachable to your NginX server?
I don't know a lot about NginX configuration, but in the error above it seems as if the server is not fully defined (client: 10.20.6.1, server: , request: ...), not sure if that is an indication for a problem.