Failed to deserialize APM server response and HTTP 502

Elastic Observability
1

I have a Java APM Agent running in my Openshift container application. My application/apm agent prints the following exception when trying to send metrics to APM Server. What is going on and where/what to fix?

tjavax.net.ssl|DEBUG|31|elastic-apm-server-reporter|2024-02-21 10:10:56.983 UTC|SSLCipher.java:1675|Plaintext after DECRYPTION (
  0000: 48 54 54 50 2F 31 2E 31   20 35 30 32 20 42 61 64  HTTP/1.1 502 Bad
  0010: 20 47 61 74 65 77 61 79   0D 0A 43 6F 6E 74 65 6E   Gateway..Conten
  0020: 74 2D 54 79 70 65 3A 20   74 65 78 74 2F 70 6C 61  t-Type: text/pla
  0030: 69 6E 3B 20 63 68 61 72   73 65 74 3D 75 74 66 2D  in; charset=utf-
  0040: 38 0D 0A 58 2D 43 6C 6F   75 64 2D 52 65 71 75 65  8..X-Cloud-Reque
  0050: 73 74 2D 49 64 3A 20 6D   6C 68 63 74 5A 43 6B 54  st-Id: mlhctZCkT
  0060: 35 75 56 35 4E 6E 6A 4A   6B 50 6F 39 51 0D 0A 58  5uV5NnjJkPo9Q..X
  0070: 2D 43 6F 6E 74 65 6E 74   2D 54 79 70 65 2D 4F 70  -Content-Type-Op
  0080: 74 69 6F 6E 73 3A 20 6E   6F 73 6E 69 66 66 0D 0A  tions: nosniff..
  0090: 58 2D 46 6F 75 6E 64 2D   48 61 6E 64 6C 69 6E 67  X-Found-Handling
  00A0: 2D 43 6C 75 73 74 65 72   3A 20 37 32 39 64 32 65  -Cluster: 729d2e
  00B0: 31 32 33 33 63 64 34 33   36 63 38 66 64 33 37 65  1233cd436c8fd37e
  00C0: 31 39 36 32 64 31 34 65   31 37 0D 0A 58 2D 46 6F  1962d14e17..X-Fo
  00D0: 75 6E 64 2D 48 61 6E 64   6C 69 6E 67 2D 49 6E 73  und-Handling-Ins
  00E0: 74 61 6E 63 65 3A 20 69   6E 73 74 61 6E 63 65 2D  tance: instance-
  00F0: 30 30 30 30 30 30 30 30   34 38 0D 0A 44 61 74 65  0000000048..Date
  0100: 3A 20 57 65 64 2C 20 32   31 20 46 65 62 20 32 30  : Wed, 21 Feb 20
  0110: 32 34 20 31 30 3A 31 30   3A 35 36 20 47 4D 54 0D  24 10:10:56 GMT.
  0120: 0A 43 6F 6E 74 65 6E 74   2D 4C 65 6E 67 74 68 3A  .Content-Length:
  0130: 20 31 32 30 0D 0A 0D 0A   50 6F 73 74 20 22 68 74   120....Post "ht
  0140: 74 70 73 3A 2F 2F 31 30   2E 31 33 32 2E 38 32 2E  tps://10.132.82.
  0150: 33 32 3A 31 38 33 39 34   2F 69 6E 74 61 6B 65 2F  32:18394/intake/
  0160: 76 32 2F 65 76 65 6E 74   73 22 3A 20 73 74 72 65  v2/events": stre
  0170: 61 6D 20 65 72 72 6F 72   3A 20 73 74 72 65 61 6D  am error: stream
  0180: 20 49 44 20 32 34 30 37   31 31 39 3B 20 49 4E 54   ID 2407119; INT
  0190: 45 52 4E 41 4C 5F 45 52   52 4F 52 3B 20 72 65 63  ERNAL_ERROR; rec
  01A0: 65 69 76 65 64 20 66 72   6F 6D 20 70 65 65 72 0A  eived from peer.
)
2024-02-21 10:10:56,984 [elastic-apm-server-reporter] WARN  co.elastic.apm.agent.report.AbstractIntakeApiHandler - Response body: Post "https://10.132.82.32:18394/intake/v2/events": stream error: stream ID 2407119; INTERNAL_ERROR; received from peer

2024-02-21 10:10:56,985 [elastic-apm-server-reporter] WARN  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - failed to deserialize APM server response
com.dslplatform.json.ParsingException: Expecting '{' for map start. Found P at position: 1, following: `P`, before: `ost "https://10.132.`
	at com.dslplatform.json.ParsingException.create(ParsingException.java:16) ~[elastic-apm-agent-1.46.0.jar:1.46.0]
	at com.dslplatform.json.JsonReader.newParseError(JsonReader.java:416) ~[elastic-apm-agent-1.46.0.jar:1.46.0]
	at com.dslplatform.json.JsonReader.newParseError(JsonReader.java:404) ~[elastic-apm-agent-1.46.0.jar:1.46.0]
	at com.dslplatform.json.ObjectConverter.deserializeMap(ObjectConverter.java:100) ~[elastic-apm-agent-1.46.0.jar:1.46.0]
	at com.dslplatform.json.ObjectConverter$2.read(ObjectConverter.java:20) ~[elastic-apm-agent-1.46.0.jar:1.46.0]
	at com.dslplatform.json.ObjectConverter$2.read(ObjectConverter.java:16) ~[elastic-apm-agent-1.46.0.jar:1.46.0]
	at com.dslplatform.json.DslJson.deserialize(DslJson.java:1551) ~[elastic-apm-agent-1.46.0.jar:1.46.0]
	at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.readAccepted(IntakeV2ReportingEventHandler.java:249) [elastic-apm-agent-1.46.0.jar:1.46.0]
	at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onConnectionError(IntakeV2ReportingEventHandler.java:236) [elastic-apm-agent-1.46.0.jar:1.46.0]
	at co.elastic.apm.agent.report.AbstractIntakeApiHandler.onRequestError(AbstractIntakeApiHandler.java:237) [elastic-apm-agent-1.46.0.jar:1.46.0]
	at co.elastic.apm.agent.report.AbstractIntakeApiHandler.endRequest(AbstractIntakeApiHandler.java:211) [elastic-apm-agent-1.46.0.jar:1.46.0]
	at co.elastic.apm.agent.report.AbstractIntakeApiHandler.endRequest(AbstractIntakeApiHandler.java:178) [elastic-apm-agent-1.46.0.jar:1.46.0]
	at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:90) [elastic-apm-agent-1.46.0.jar:1.46.0]
	at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:39) [elastic-apm-agent-1.46.0.jar:1.46.0]
	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) [elastic-apm-agent-1.46.0.jar:1.46.0]
	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) [elastic-apm-agent-1.46.0.jar:1.46.0]
	at co.elastic.apm.agent.util.ExecutorUtils$2.run(ExecutorUtils.java:99) [elastic-apm-agent-1.46.0.jar:1.46.0]
	at java.lang.Thread.run(Thread.java:840) [?:?]
2024-02-21 10:10:56,989 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.AbstractIntakeApiHandler - Backing off for 0 seconds (+/-10%)
2024-02-21 10:10:56,989 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.AbstractIntakeApiHandler - Error sending data to APM server: Server returned HTTP response code: 502 for URL: https://729d2e1233cd436c8fd37e1962d14e17.ece-test.acme.com:9243/intake/v2/events, response code is 502

Any directions would be helpful! Thanks,
Andreas