Kibana version:
7.4.0
Elasticsearch version:
7.4.0
APM Server version:
7.4.0
APM Agent language and version:
Java, 1.11.0
Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.
my apm-server is on Kubernetes, and has nginx-proxy in front.
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
The elastic-apm-agent and the java application reside on my local system, I am able to send logs when I run kubectl port forward apm-server 8200 but I get 502 status when I specify nginx-proxy loadbalancer url in elasticapm.properties file.
Provide logs and/or server output (if relevant):
my elasticapm.properties file looks like this:
service_name=demo
application_packages=ca.com.elastic
server_urls=https://172.20.11.188
verify_server_cert=false
log_level=debug
apm-server.yaml file:
apm-server.yml: |
host: "0.0.0.0:8200"
setup.template.settings:
index:
number_of_shards: 1
codec: best_compression
setup.kibana:
host: "< kibana endpoint >"
output.elasticsearch:
hosts: ["elasticsearch:9200"]
logs from elastic-apm-agent:
2019-11-22 11:25:03.188 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Starting new request to https://172.20.11.188/intake/v2/events
502 Bad Gateway
2019-11-22 11:25:03.224 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Scheduling request timeout in 10s
2019-11-22 11:25:13.225 [apm-request-timeout-timer] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Request flush because the request timeout occurred
2019-11-22 11:25:13.235 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving FLUSH event (sequence 1)
2019-11-22 11:25:13.238 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Flushing 1386 uncompressed 601 compressed bytes
2019-11-22 11:25:13.240 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 0 seconds (+/-10%)
2019-11-22 11:25:13.240 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Server returned HTTP response code: 502 for URL: https://172.20.11.188/intake/v2/events, response code is 502
2019-11-22 11:25:13.240 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Sending payload to APM server failed
java.io.IOException: Server returned HTTP response code: 502 for URL: https://172.20.11.188/intake/v2/events
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.flush(IntakeV2ReportingEventHandler.java:224)
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.handleEvent(IntakeV2ReportingEventHandler.java:128)
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:116)
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:50)
at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
at java.lang.Thread.run(Thread.java:748)
2019-11-22 11:25:13.243 [apm-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler -502 Bad Gateway
nginx/1.13.12
Logs from nginx proxy pod:
2019/11/22 19:24:35 [error] 10#10: *309 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.6.1, server: , request: "POST /config/v1/agents HTTP/1.1", upstream: "http://10.19.72.106:8200/config/v1/agents", host: "172.20.11.188"
10.20.6.1 - - [22/Nov/2019:19:24:35 +0000] "POST /config/v1/agents HTTP/1.1" 502 174 "-" "elasticapm-java/1.11.0" "-"
2019/11/22 19:24:44 [info] 10#10: *309 client 10.20.6.1 closed keepalive connection
2019/11/22 19:25:43 [error] 10#10: *311 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.6.1, server: , request: "POST /intake/v2/events HTTP/1.1", upstream: "http://10.19.72.106:8200/intake/v2/events", host: "172.20.11.188"
10.20.6.1 - - [22/Nov/2019:19:25:43 +0000] "POST /intake/v2/events HTTP/1.1" 502 174 "-" "elasticapm-java/1.11.0" "-"
2019/11/22 19:25:49 [info] 10#10: *311 client 10.20.6.1 closed keepalive connection
2019/11/22 19:26:13 [error] 10#10: *313 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.6.1, server: , request: "POST /intake/v2/events HTTP/1.1", upstream: "http://10.19.72.106:8200/intake/v2/events", host: "172.20.11.188"
10.20.6.1 - - [22/Nov/2019:19:26:13 +0000] "POST /intake/v2/events HTTP/1.1" 502 174 "-" "elasticapm-java/1.11.0" "-"
2019/11/22 19:26:26 [info] 10#10: *313 client 10.20.6.1 closed keepalive connection
10.20.6.1 - - [22/Nov/2019:19:27:13 +0000] "POST /intake/v2/events HTTP/1.1" 502 174 "-" "elasticapm-java/1.11.0" "-"
2019/11/22 19:27:13 [error] 10#10: *315 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.6.1, server: , request: "POST /intake/v2/events HTTP/1.1", upstream: "http://10.19.72.106:8200/intake/v2/events", host: "172.20.11.188"
2019/11/22 19:27:33 [info] 10#10: *315 client 10.20.6.1 closed keepalive connection
2019/11/22 19:27:43 [error] 10#10: *317 connect() failed (111: Connection refused) while connecting to upstream, client: 10.20.6.1, server: , request: "POST /intake/v2/events HTTP/1.1", upstream: "http://10.19.72.106:8200/intake/v2/events", host: "172.20.11.188"
Any ideas?