Java Agent APM won't send data to Elasticsearch

[zerratriani]

I have a java application and want to monitor it. the two hosts between the APM server and the application server are well connected. I have also deactivated the firewall but the data from the apm agent was not successfully sent to elasticsearch and did not appear in kibana

Kibana version: 8.1.0

Elasticsearch version: 8.1.0

APM Server version: 8.1.0

APM Agent language and version: JAVA version 1.35.0

Log from APM-Server:

{"log.level":"error","@timestamp":"2023-01-19T03:21:45.919-0500","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: EOF","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-19T03:22:00.930-0500","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: EOF","service.name":"apm-server","ecs.version":"1.6.0"}

Log from Catalina.out:

2023-01-19 15:38:03,543 [elastic-apm-server-reporter] WARN  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - null
2023-01-19 15:38:03,543 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)
2023-01-19 15:38:38,979 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Read timed out, response code is -1

Configuration in Catalina.sh

export CATALINA_OPTS="$CATALINA_OPTS -javaagent:/opt/Elastic/elastic-apm.jar -Delastic.apm.service_name=supercar -Delastic.apm.server_urls=http://172.16.11.226:8200 -Delastic.apm.environment=production -Delastic.apm.application_packages=org.apache.org"

And here if i curl from application server to APM Server

[Sylvain_Juge]

Hi @zerratriani ,

Could you capture the logs using this procedure please ?

For now only the agent logs are required for investigation.

[simitt (Silvia Mitter)]

This part of the APM Server logs indicates that something is not right with the setup for the apm-server, as it cannot verify that the index templates are properly setup.

@zerratriani

precondition failed: EOF

Please provide more details how you set up the APM Server (standalone or managed by Elastic Agent & Fleet) and if you have changed Elasticsearch output configurations in the APM Server.