Our Java agent could not connect to APM server, seeing the following error
2022-02-08 12:13:32,519 [elastic-apm-shared] DEBUG co.elastic.apm.agent.report.ApmServerReporter - Could not add JsonWriter {"metricset":{"timestamp":1644340412519000,"tags":{"name":"scavenge"},"samples":{"jvm.gc.time":{"value":14563.0},"jvm.gc.count":{"value":64.0}}}}
to ring buffer as no slots are available
2022-02-08 12:13:32,519 [elastic-apm-shared] DEBUG co.elastic.apm.agent.report.ApmServerReporter - Could not add JsonWriter {"metricset":{"timestamp":1644340412519000,"tags":{"name":"nursery-survivor"},"samples":{"jvm.memory.heap.pool.committed":{"value":433651712.0},"jvm.memory.heap.pool.max":{"value":590774272.0},"jvm.memory.heap.pool.used":{"value":433651712.0}}}}
to ring buffer as no slots are available
2022-02-08 12:13:32,535 [elastic-apm-shared] DEBUG co.elastic.apm.agent.report.ApmServerReporter - Could not add JsonWriter {"metricset":{"timestamp":1644340412519000,"samples":{"system.process.cpu.total.norm.pct":{"value":0.008072681213464607},"jvm.memory.heap.used":{"value":4257639384.0},"jvm.memory.non_heap.used":{"value":625746132.0},"jvm.memory.heap.max":{"value":17179869184.0},"jvm.memory.non_heap.committed":{"value":993469124.0},"system.cpu.total.norm.pct":{"value":0.13332427928152732},"system.process.memory.size":{"value":7151595520.0},"jvm.memory.heap.committed":{"value":6026297344.0},"jvm.memory.non_heap.max":{"value":-1.0},"system.memory.actual.free":{"value":11020980224.0},"system.memory.total":{"value":25769332736.0},"jvm.thread.count":{"value":270.0}}}}
to ring buffer as no slots are available
2022-02-08 12:13:32,535 [elastic-apm-shared] DEBUG co.elastic.apm.agent.report.ApmServerReporter - Could not add JsonWriter {"metricset":{"timestamp":1644340412535000,"transaction":{"name":"LeadServicesApi#retrieveLead","type":"request"},"span":{"type":"app","subtype":null},"tags":{},"samples":{"span.self_time.count":{"value":3},"span.self_time.sum.us":{"value":12273}}}}
to ring buffer as no slots are available
2022-02-08 12:13:32,535 [elastic-apm-shared] DEBUG co.elastic.apm.agent.report.ApmServerReporter - Could not add JsonWriter {"metricset":{"timestamp":1644340412535000,"transaction":{"name":"HealthCheckApi#getProdConfigStatus","type":"request"},"span":{"type":"app","subtype":null},"tags":{},"samples":{"span.self_time.count":{"value":3},"span.self_time.sum.us":{"value":6718}}}}
to ring buffer as no slots are available
2022-02-08 12:13:32,535 [elastic-apm-shared] DEBUG co.elastic.apm.agent.report.ApmServerReporter - Could not add JsonWriter {"metricset":{"timestamp":1644340412535000,"transaction":{"name":"FormContentRestApi#getFormContentByName","type":"request"},"span":{"type":"app","subtype":null},"tags":{},"samples":{"span.self_time.count":{"value":3},"span.self_time.sum.us":{"value":14679}}}}
to ring buffer as no slots are available
2022-02-08 12:13:32,535 [elastic-apm-shared] DEBUG co.elastic.apm.agent.report.ApmServerReporter - Could not add JsonWriter {"metricset":{"timestamp":1644340412535000,"tags":{"name":"tenured-LOA"},"samples":{"jvm.memory.heap.pool.committed":{"value":143681536.0},"jvm.memory.heap.pool.max":{"value":832150528.0},"jvm.memory.heap.pool.used":{"value":0.0}}}}
to ring buffer as no slots are available
2022-02-08 12:13:32,535 [elastic-apm-shared] DEBUG co.elastic.apm.agent.report.ApmServerReporter - Could not add JsonWriter {"metricset":{"timestamp":1644340412535000,"tags":{"name":"tenured-SOA"},"samples":{"jvm.memory.heap.pool.committed":{"value":2729940992.0},"jvm.memory.heap.pool.max":{"value":15810847744.0},"jvm.memory.heap.pool.used":{"value":2091991704.0}}}}
to ring buffer as no slots are available
2022-02-08 12:13:32,535 [elastic-apm-shared] DEBUG co.elastic.apm.agent.report.ApmServerReporter - Could not add JsonWriter {"metricset":{"timestamp":1644340412535000,"tags":{"name":"nursery-allocate"},"samples":{"jvm.memory.heap.pool.committed":{"value":2719023104.0},"jvm.memory.heap.pool.max":{"value":3704193024.0},"jvm.memory.heap.pool.used":{"value":1732073792.0}}}}
to ring buffer as no slots are available
2022-02-08 12:13:32,535 [elastic-apm-shared] DEBUG co.elastic.apm.agent.report.ApmServerReporter - Could not add JsonWriter {"metricset":{"timestamp":1644340412535000,"tags":{"name":"global"},"samples":{"jvm.gc.time":{"value":5148.0},"jvm.gc.count":{"value":8.0}}}}
to ring buffer as no slots are available
2022-02-08 12:13:32,535 [elastic-apm-shared] DEBUG co.elastic.apm.agent.report.ApmServerReporter - Could not add JsonWriter {"metricset":{"timestamp":1644340412535000,"transaction":{"name":"HealthCheck#checkServicesState","type":"request"},"span":{"type":"app","subtype":null},"tags":{},"samples":{"span.self_time.count":{"value":3},"span.self_time.sum.us":{"value":2283426}}}}
to ring buffer as no slots are available
2022-02-08 12:13:34,303 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error trying to connect to APM Server at http://xxxxxxx:8200/intake/v2/events. Some details about SSL configurations corresponding the current connection are logged at INFO level.
2022-02-08 12:13:34,303 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type JSON_WRITER with this error: connect timed out
2022-02-08 12:13:34,303 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Event handling failure
java.net.SocketTimeoutException: connect timed out
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:236) ~[?:1.8.0]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:218) ~[?:1.8.0]
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:183) ~[?:1.8.0]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) ~[?:1.8.0]
at java.net.Socket.connect(Socket.java:682) ~[?:1.8.0]
at sun.net.NetworkClient.doConnect(NetworkClient.java:187) ~[?:1.8.0]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:494) ~[?:1.8.0]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:589) ~[?:1.8.0]
at sun.net.www.http.HttpClient.(HttpClient.java:256) ~[?:1.8.0]
at sun.net.www.http.HttpClient.New(HttpClient.java:360) ~[?:1.8.0]
at sun.net.www.http.HttpClient.New(HttpClient.java:378) ~[?:1.8.0]
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1238) ~[?:1.8.0]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1174) ~[?:1.8.0]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1068) ~[?:1.8.0]
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1002) ~[?:1.8.0]
at co.elastic.apm.agent.report.AbstractIntakeApiHandler.startRequest(AbstractIntakeApiHandler.java:113) ~[elastic-apm-agent-1.28.4.jar:?]
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.startRequest(IntakeV2ReportingEventHandler.java:165) [elastic-apm-agent-1.28.4.jar:?]
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.handleIntakeEvent(IntakeV2ReportingEventHandler.java:112) [elastic-apm-agent-1.28.4.jar:?]
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.dispatchEvent(IntakeV2ReportingEventHandler.java:98) [elastic-apm-agent-1.28.4.jar:?]
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:69) [elastic-apm-agent-1.28.4.jar:?]
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:36) [elastic-apm-agent-1.28.4.jar:?]
at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) [elastic-apm-agent-1.28.4.jar:?]
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) [elastic-apm-agent-1.28.4.jar:?]
at java.lang.Thread.run(Thread.java:825) [?:2.9 (06-22-2021)]
2022-02-08 12:13:34,303 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)
Kibana version: 7:17
Elasticsearch version:7:17
APM Server version:7:17
APM Agent language and version: Java agent version 1.28.4
Browser version:
Original install method (e.g. download page, yum, deb, from source, etc.) and version:yum
Fresh install or upgraded from other version?: uPGRADE FROM 7.16
Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc. No
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
Steps to reproduce:
1.
2.
3.
Errors in browser console (if relevant):
Provide logs and/or server output (if relevant):