Sending payload to APM server failed

Elastic Observability APM
1

Kibana version: 7.9.2

Elasticsearch version: 7.9.2

APM Server version: 7.9.2

APM Agent language and version: Java agent - 1.18 / 1.9.0

Browser version:

Original install method (e.g. download page, yum, deb, from source, etc.) and version: rpm

Fresh install or upgraded from other version? fresh install

Is there anything special in your setup?
Nothing special, just manual attachment of the java agent to the JVM

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant): I run a java application on jboss-fuse-6.0.0.redhat-024, i had set up the apm-server, elasticsearch and kibana servers successfully, attached the apm-agent manually and can see metrics being generated. The agent is able to connect to the apm-server for healthchecks but unable to send /intake/v2/events to the apm-server. The calls never make it to the apm-server, i have enabled debugs, played with the different time out settings both on the apm-server and the agent but it still fails. Interestingly using tcpdump i noticed the agent never really sends any post out the app-server to the apm-server

Steps to reproduce:

  1. run a jboss fuse 6.0 app
  2. install the apm-agents/apm-server and instrument the agent manually
  3. check to see if events make it to the apm-server

Errors in browser console (if relevant): Full APM Debug log is attached

Provide logs and/or server output (if relevant):
key suspect errors from the apm-log
2020-10-05 02:32:42,268 [main] ERROR co.elastic.apm.agent.util.DependencyInjectingServiceLoader - unable to instantiate 'co.elastic.apm.agent.httpclient.HttpClientInstrumentation', unsupported class version error: co/elastic/apm/agent/httpclient/HttpClientInstrumentation has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0
full Debug log is below, couldn't find the upload button

  2020-10-05 02:32:37,124 [elastic-apm-init-instrumentation-shutdown-hook] INFO  co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to STOPPED state
    2020-10-05 02:32:37,124 [elastic-apm-init-instrumentation-shutdown-hook] INFO  co.elastic.apm.agent.report.ApmServerReporter - dropped events because of full queue: 0
    2020-10-05 02:32:42,133 [main] DEBUG co.elastic.apm.agent.impl.payload.SystemInfo - Could not parse container ID from '/proc/self/cgroup' line: 11:hugetlb:/
    2020-10-05 02:32:42,135 [main] DEBUG co.elastic.apm.agent.impl.payload.SystemInfo - Could not parse container ID from '/proc/self/cgroup' line: 10:blkio:/
    2020-10-05 02:32:42,135 [main] DEBUG co.elastic.apm.agent.impl.payload.SystemInfo - Could not parse container ID from '/proc/self/cgroup' line: 9:cpuset:/
    2020-10-05 02:32:42,136 [main] DEBUG co.elastic.apm.agent.impl.payload.SystemInfo - Could not parse container ID from '/proc/self/cgroup' line: 8:cpuacct,cpu:/
    2020-10-05 02:32:42,136 [main] DEBUG co.elastic.apm.agent.impl.payload.SystemInfo - Could not parse container ID from '/proc/self/cgroup' line: 7:pids:/
    2020-10-05 02:32:42,136 [main] DEBUG co.elastic.apm.agent.impl.payload.SystemInfo - Could not parse container ID from '/proc/self/cgroup' line: 6:devices:/
    2020-10-05 02:32:42,137 [main] DEBUG co.elastic.apm.agent.impl.payload.SystemInfo - Could not parse container ID from '/proc/self/cgroup' line: 5:memory:/
    2020-10-05 02:32:42,137 [main] DEBUG co.elastic.apm.agent.impl.payload.SystemInfo - Could not parse container ID from '/proc/self/cgroup' line: 4:perf_event:/
    2020-10-05 02:32:42,137 [main] DEBUG co.elastic.apm.agent.impl.payload.SystemInfo - Could not parse container ID from '/proc/self/cgroup' line: 3:net_prio,net_cls:/
    2020-10-05 02:32:42,138 [main] DEBUG co.elastic.apm.agent.impl.payload.SystemInfo - Could not parse container ID from '/proc/self/cgroup' line: 2:freezer:/
    2020-10-05 02:32:42,138 [main] DEBUG co.elastic.apm.agent.impl.payload.SystemInfo - Could not parse container ID from '/proc/self/cgroup' line: 1:name=systemd:/user.slice/user-1001.slice/session-8510.scope
    2020-10-05 02:32:42,138 [main] DEBUG co.elastic.apm.agent.impl.payload.SystemInfo - container ID is null
    2020-10-05 02:32:42,182 [main] INFO  co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.sun.management.OperatingSystemMXBean
    2020-10-05 02:32:42,254 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.18.0 as greybag on Java 1.8.0_262 Runtime version: 1.8.0_262-b10 VM version: 25.262-b10 (Oracle Corporation) Linux 3.10.0-1127.13.1.el7.x86_64
    2020-10-05 02:32:42,254 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - service_name: 'greybag' (source: Java System Properties)
    2020-10-05 02:32:42,254 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - service_node_name: 'rbxnode1' (source: Java System Properties)
    2020-10-05 02:32:42,254 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - capture_body: 'ALL' (source: Java System Properties)
    2020-10-05 02:32:42,254 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - server_urls: 'http://xx.xx.xx.xx:8200' (source: Java System Properties)
    2020-10-05 02:32:42,254 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - server_timeout: '59s' (source: Java System Properties)
    2020-10-05 02:32:42,254 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - api_request_time: '55s' (source: Java System Properties)
    2020-10-05 02:32:42,254 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - application_packages: 'com.friendlydigits.*' (source: Java System Properties)
    2020-10-05 02:32:42,254 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - log_level: 'TRACE' (source: Java System Properties)
    2020-10-05 02:32:42,254 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - log_file: '/greybag/apm/elastic-apm.log' (source: Java System Properties)
    2020-10-05 02:32:42,255 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - No plugins dir
    2020-10-05 02:32:42,268 [main] ERROR co.elastic.apm.agent.util.DependencyInjectingServiceLoader - unable to instantiate 'co.elastic.apm.agent.httpclient.HttpClientInstrumentation', unsupported class version error: co/elastic/apm/agent/httpclient/HttpClientInstrumentation has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0
    2020-10-05 02:32:42,268 [main] ERROR co.elastic.apm.agent.util.DependencyInjectingServiceLoader - unable to instantiate 'co.elastic.apm.agent.httpclient.HttpClientAsyncInstrumentation', unsupported class version error: co/elastic/apm/agent/httpclient/HttpClientAsyncInstrumentation has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0
    2020-10-05 02:32:42,269 [main] ERROR co.elastic.apm.agent.util.DependencyInjectingServiceLoader - unable to instantiate 'co.elastic.apm.agent.httpclient.HttpRequestHeadersInstrumentation', unsupported class version error: co/elastic/apm/agent/httpclient/HttpRequestHeadersInstrumentation has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0
    2020-10-05 02:32:42,301 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying instrumentation co.elastic.apm.agent.httpclient.ApacheHttpClientInstrumentation
    2020-10-05 02:32:42,333 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying instrumentation co.elastic.apm.agent.httpclient.ApacheHttpAsyncClientInstrumentation
    2020-10-05 02:32:42,352 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying instrumentation co.elastic.apm.agent.plugin.api.ElasticApmApiInstrumentation$StartTransactionWithRemoteParentInstrumentation
 2020-10-05 02:32:42,398 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying instrumentation co.elastic.apm.agent.concurrent.ForkJoinTaskInstrumentation
    2020-10-05 02:32:42,405 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying instrumentation co.elastic.apm.agent.jaxrs.JaxRsTransactionNameInstrumentation
    2020-10-05 02:32:42,406 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying instrumentation co.elastic.apm.agent.jaxws.JaxWsTransactionNameInstrumentation
    2020-10-05 02:32:42,406 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying instrumentation co.elastic.apm.agent.jdbc.ConnectionInstrumentation
    {"metricset":{"timestamp":1601861594314000,"samples":{"system.process.cpu.total.norm.pct":{"value":0.7383071148412129},"jvm.memory.heap.used":{"value":1520500736.0},"jvm.memory.non_heap.used":{"value":251118648.0},"jvm.memory.heap.max":{"value":3817865216.0},"jvm.gc.alloc":{"value":7664587752.0},"jvm.memory.non_heap.committed":{"value":288817152.0},"system.cpu.total.norm.pct":{"value":0.7297787144362486},"system.process.cgroup.memory.mem.usage.bytes":{"value":35433328640.0},"system.process.memory.size":{"value":11409514496.0},"system.process.cgroup.memory.stats.inactive_file.bytes":{"value":493170688.0},"jvm.memory.heap.committed":{"value":3028811776.0},"jvm.memory.non_heap.max":{"value":-1.0},"system.memory.actual.free":{"value":62315937792.0},"system.memory.total":{"value":67368673280.0},"jvm.thread.count":{"value":954.0}}}}
    {"metricset":{"timestamp":1601861594314000,"tags":{"name":"PS Survivor Space"},"samples":{"jvm.memory.heap.pool.committed":{"value":67108864.0},"jvm.memory.heap.pool.max":{"value":67108864.0},"jvm.memory.heap.pool.used":{"value":25015736.0}}}}

    2020-10-05 02:34:03,919 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 0 seconds (+/-10%)
    2020-10-05 02:34:03,919 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Error writing request body to server, response code is -1
    2020-10-05 02:34:03,919 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Sending payload to APM server failed
    java.io.IOException: Error writing request body to server
            at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3597) ~[?:1.8.0_262]
            at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3580) ~[?:1.8.0_262]
            at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:253) ~[?:1.8.0_262]
            at java.util.zip.DeflaterOutputStream.finish(DeflaterOutputStream.java:226) ~[?:1.8.0_262]
            at java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:238) ~[?:1.8.0_262]
            at co.elastic.apm.agent.report.AbstractIntakeApiHandler.endRequest(AbstractIntakeApiHandler.java:156) [?:?]
            at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.endRequest(IntakeV2ReportingEventHandler.java:172) [?:?]
            at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.handleEvent(IntakeV2ReportingEventHandler.java:85) [?:?]
            at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:73) [?:?]
            at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:44) [?:?]
            at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) [?:?]
            at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) [?:?]
            at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262]
    2020-10-05 02:34:03,919 [elastic-apm-server-reporter] WARN  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - null
    2020-10-05 02:34:05,067 [elastic-apm-circuit-breaker] TRACE co.elastic.apm.agent.impl.circuitbreaker.CircuitBreaker - Scheduling next stress monitor polling in 5000s
2

Hi,

Others have experienced a similar error(like here) and the solution was that the error is not an error and will be demoted to a debug message in later releases.

The real error seems to be this:

Did you have a look in the logs on the APM server?
Do you have a firewall in between(although you already said that the agent can connect for healthchecks)?

Best regards
Wolfram

1 Like
3

Yes I did, i have debug enabled on the APM Server too. I see the healthChecks GET Requests come in and the response. but the POST /intake/v2/events never come to it.
The apm-server and the apm-agent system are on the same LAN, there's no firewall between them.
Again using tcpdump to inspect the packets, i can also see healthchecks request /response on both servers, but while the apm-agent says it has made the POST event calls it doesn't show at all on the tcp dump that it tried to call the apm-server. That is the strange effect.

when i user CURL make a POST event to the apm-server, i can see the request attempt on the tcpdump and on the apm-server even though it fails with validation/content-type errors.

4

Maybe there's a proxy between your application and APM Server that's terminating the connection?

By default, the agent keeps the HTTP request open for 10s to streams data through the same request. It might run into a timeout.

Try reducing api_request_time and possibly api_request_size or re-configure your proxy or network settings to allow for longer requests.

I also can't see a healthcheck message in the agent logs. There are supposed to be two debug logs:

Starting healthcheck to {}
APM server {} version is: {}

5

Thank you @felixbarny,
There is no proxy or firewall between the servers - they are on the same LAN.
I have increased both the api_request_time (up to a minute) but the experience is still the same. You might have something on the api_request_size.
When i run the agent with the default settings, the failure reason is read timed out but when i tweaked the api_request_time and api_request_size, then the error changes to java.io.IOException: Error writing request body to server is there a way to check the size of the events being generated?

Please find below the healthchecks requests and response to and from the APM Server

2020-10-05 02:32:44,462 [elastic-apm-server-healthcheck] DEBUG co.elastic.apm.agent.report.ApmServerHealthChecker - Starting healthcheck to http://xx.xx.xx.xx:8200/
2020-10-05 02:32:44,462 [elastic-apm-remote-config-poller] DEBUG co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Reloading configuration from APM Server http://xx.xx.xx.xx:8200/config/v1/agents
2020-10-05 02:32:44,484 [elastic-apm-server-healthcheck] DEBUG co.elastic.apm.agent.bci.IndyPluginClassLoaderFactory - Creating plugin class loader for null containing [co.elastic.apm.agent.urlconnection.HttpUrlConnectionInstrumentation, co.elastic.apm.agent.urlconnection.HttpUrlConnectionInstrumentation$DisconnectInstrumentation, co.elastic.apm.agent.urlconnection.HttpUrlConnectionInstrumentation$CreateSpanInstrumentation, co.elastic.apm.agent.urlconnection.UrlConnectionPropertyAccessor, co.elastic.apm.agent.urlconnection.SSLContextInstrumentation, co.elastic.apm.agent.urlconnection.package-info]
2020-10-05 02:32:44,506 [elastic-apm-remote-config-poller] TRACE co.elastic.apm.agent.report.serialize.DslJsonSerializer - {"service":{"name":"greybound","agent":{"name":"java","ephemeral_id":"b3943c6e-d97b-4344-86e7-ab72254ae622","version":"1.18.0"},"language":{"name":"Java","version":"1.8.0_262"},"node":{"configured_name":"rbxnode1"},"runtime":{"name":"Java","version":"1.8.0_262"},"version":null},"process":{"pid":36351,"title":"/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre/bin/java"},"system":{"architecture":"amd64","hostname":"ungcorgreybound01","platform":"Linux"}}
2020-10-05 02:32:44,511 [elastic-apm-remote-config-poller] DEBUG co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Central configuration is disabled. Set kibana.enabled: true in your APM Server configuration.
2020-10-05 02:32:44,513 [elastic-apm-remote-config-poller] DEBUG co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Scheduling next remote configuration reload in 300s
2020-10-05 02:32:44,513 [elastic-apm-server-healthcheck] INFO  co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: {  "build_date": "2020-09-22T22:02:35Z",  "build_sha": "34f81a2208c862777ca587867fd3aa2373c3f669",  "version": "7.9.2"}
2020-10-05 02:32:44,515 [elastic-apm-server-healthcheck] DEBUG co.elastic.apm.agent.report.ApmServerHealthChecker - APM server http://xx.xx.xx.xx:8200/ version is: 7.9.2
6

My suggestion would be do decrease, not increase the values.

See also Common problems | APM Server Reference [7.15] | Elastic for some more backgorund.

After a successful flush, the agent logs the size Flushing {} uncompressed {} compressed bytes. Currently, we don't log that if an error occurs.
It would be relatively easy to add that in co.elastic.apm.agent.report.AbstractIntakeApiHandler#onRequestError by logging deflater.getBytesWritten().

7

Thanks @felixbarny for the support, it is truly appreciated.
that means running with the default timeout settings right? or less than that - say 5s for the api_request_time?

Again how does this translate to the actual communication on the wire, whether a timeout occurs or it is not able to write the events, would it not show up on the tcpdump that an attempt to call the apmserver was made?

Please find below the tcpdump between the agent and the server... you can see when the healthcheck and agent configuration calls are made to the server but nothing shows up for the post /intake/v2/events and that is my concern... if i was seeing the calls to post the event and then the errors occurs, then that would make more sense to me

or is there something that must happen on the agent system before actually calling the apm-server?

here is the tcpdump mentioned

$ sudo tcpdump -i "ens192" -nn -A dst "xx.xx.xx.xx" and port 8200 -vv
tcpdump: listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
11:04:14.202091 IP (tos 0x0, ttl 64, id 14188, offset 0, flags [DF], proto TCP (6), length 60)
    aa.aa.aa.aa.35274 > xx.xx.xx.xx.8200: Flags [S], cksum 0xb2dc (incorrect -> 0x967d), seq 2390813267, win 29200, options [mss 1460,sackOK,TS val 2363350335 ecr 0,nop,wscale 7], length 0
E..<7l@.@.P.
..d
..u.. ....S......r............
...?........
11:04:14.202694 IP (tos 0x0, ttl 64, id 14189, offset 0, flags [DF], proto TCP (6), length 52)
    aa.aa.aa.aa.35274 > xx.xx.xx.xx.8200: Flags [.], cksum 0xb2d4 (incorrect -> 0xe7e2), seq 2390813268, ack 4089025105, win 229, options [nop,nop,TS val 2363350335 ecr 50513268], length 0
E..47m@.@.P.
..d
..u.. ....T...Q...........
...?...t
11:04:14.206006 IP (tos 0x0, ttl 64, id 14190, offset 0, flags [DF], proto TCP (6), length 219)
    aa.aa.aa.aa.35274 > xx.xx.xx.xx.8200: Flags [P.], cksum 0xb37b (incorrect -> 0xf87a), seq 0:167, ack 1, win 229, options [nop,nop,TS val 2363350339 ecr 50513268], length 167
E...7n@.@.P.
..d
..u.. ....T...Q.....{.....
...C...tGET / HTTP/1.1
User-Agent: elasticapm-java/1.18.0
Host: xx.xx.xx.xx:8200
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive


11:04:14.206886 IP (tos 0x0, ttl 64, id 14191, offset 0, flags [DF], proto TCP (6), length 52)
    aa.aa.aa.aa.35274 > xx.xx.xx.xx.8200: Flags [.], cksum 0xb2d4 (incorrect -> 0xe641), seq 167, ack 234, win 237, options [nop,nop,TS val 2363350340 ecr 50513272], length 0
E..47o@.@.P.
..d
..u.. ........:...........
...D...x
11:04:14.212526 IP (tos 0x0, ttl 64, id 14192, offset 0, flags [DF], proto TCP (6), length 289)
    aa.aa.aa.aa.35274 > xx.xx.xx.xx.8200: Flags [P.], cksum 0xb3c1 (incorrect -> 0xd823), seq 167:404, ack 234, win 237, options [nop,nop,TS val 2363350345 ecr 50513272], length 237
E..!7p@.@.O.
..d
..u.. ........:...........
...I...xPOST /config/v1/agents HTTP/1.1
User-Agent: elasticapm-java/1.18.0
Content-Type: application/json
Host: xx.xx.xx.xx:8200
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Content-Length: 465


11:04:14.212583 IP (tos 0x0, ttl 64, id 14193, offset 0, flags [DF], proto TCP (6), length 517)
    aa.aa.aa.aa.35274 > xx.xx.xx.xx.8200: Flags [P.], cksum 0xb4a5 (incorrect -> 0xdf3e), seq 404:869, ack 234, win 237, options [nop,nop,TS val 2363350345 ecr 50513272], length 465
E...7q@.@.N.
..d
..u.. ........:...........
...I...x{"service":{"name":"greybound","agent":{"name":"java","ephemeral_id":"8348190b-6467-4827-8de3-a4968ced43c2","version":"1.18.0"},"language":{"name":"Java","version":"1.8.0_262"},"node":{"configured_name":"rbxnode1"},"runtime":{"name":"Java","version":"1.8.0_262"},"version":null},"process":{"pid":47719,"title":"/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre/bin/java"},"system":{"architecture":"amd64","hostname":"ungcorgreybound01","platform":"Linux"}}
11:04:14.291797 IP (tos 0x0, ttl 64, id 14194, offset 0, flags [DF], proto TCP (6), length 52)
    aa.aa.aa.aa.35274 > xx.xx.xx.xx.8200: Flags [.], cksum 0xb2d4 (incorrect -> 0xe21a), seq 869, ack 456, win 245, options [nop,nop,TS val 2363350425 ecr 50513318], length 0
E..47r@.@.P.
..d
..u.. ....................
........
11:04:24.210837 IP (tos 0x0, ttl 64, id 14195, offset 0, flags [DF], proto TCP (6), length 52)
    aa.aa.aa.aa.35274 > xx.xx.xx.xx.8200: Flags [F.], cksum 0xb2d4 (incorrect -> 0xbb5a), seq 869, ack 456, win 245, options [nop,nop,TS val 2363360344 ecr 50513318], length 0
E..47s@.@.P.
..d
..u.. ....................
...X....
11:04:24.211596 IP (tos 0x0, ttl 64, id 14196, offset 0, flags [DF], proto TCP (6), length 52)
    aa.aa.aa.aa.35274 > xx.xx.xx.xx.8200: Flags [.], cksum 0xb2d4 (incorrect -> 0x9471), seq 870, ack 457, win 245, options [nop,nop,TS val 2363360344 ecr 50523278], length 0
E..47t@.@.P.
..d
..u.. ....................
...X....
8

Try with 1s, 2s, 5s.

Maybe there are issues with chunked-encoded HTTP requests in your system. Either that they are not working at all or the chunk length the Java agent uses (16k) causes problems.
Looking at the stacktrace, the error happens when flushing the data before closing the HTTP connection.

We set the buffer size here:

You could try setting a lower buffer size, like 5 * 1024 or 10 * 1024 and see if that helps.

But there's definitely something strange going on in your system.

9

Thank you @felixbarny,
will try to get one of my colleagues in java development to assist me with this.

Kind regards

10

Hello @felixbarny,
Further troubleshooting, we installed an apm-server on the same host as the agent. The agent is able to post events to the apm-server when we used localhost but when we changed the address from localhost to the ip address we ran into the same challenges.

#p.s struggling with buffersize settings.

regards

11

Hi @felixbarny, no further ideas on my feedback of the apm events working for a localhost and failing when changed to an ip address?

12

Seems like there's an issue with the network interface, not with the agent itself.

13

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.