Apm-server not work with jboss 7

Hi
setup apm-agent java for jboss server

-jboss 7
-elastic-apm-agent-1.35.0.jar
-elasticsearch-8.5.3

here is the apm-server.yml

vi /etc/apm-server/apm-server.yml

output.elasticsearch:
hosts: ["localhost:9200"]
username: $MYUSER
password: $MYPASS

#APM Server
host: ":8200"

on apm server listen to 8200:
netstat -tpuln

tcp 0 0 0.0.0.0:8200 0.0.0.0:* LISTEN 738822/apm-server
tcp 0 0 0.0.0.0:9200 0.0.0.0:* LISTEN 128891/java

here is the jboss:

<jvm>
    <jvm-options>
        <option value="-javaagent:/use/app/elastic-apm-agent-1.35.0.jar"/>
        <option value="-Delastic.apm.service_name=myapp"/>
        <option value="-Delastic.apm.server_urls=http://$ELASTICSERVERIP:8200"/>
        <option value="-Delastic.apm.secret_token=$MYTOKEN"/>
        <option value="-Delastic.apm.environment=development"/>
        <option value="-Delastic.apm.application_packages=$MYPACKAGE"/>
    </jvm-options>
</jvm>

on elastic server doesn't show anything from apm-agent!

FYI: I run tcptump on ELASTIC but noting send from apm-agent!

also enable debug mode, here is the log:


[Server:main-server] 2022-12-28 16:10:03,888 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 28)
[Server:main-server] 2022-12-28 16:10:03,889 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.util.UrlConnectionUtils - Opening http://ELASTICKSERVER:8200/intake/v2/events without proxy
[Server:main-server] 2022-12-28 16:10:03,889 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Starting new request to http://ELASTICSERVER:8200/intake/v2/events
[Server:main-server] 2022-12-28 16:10:03,892 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Scheduling request timeout in 10s
[Server:main-server] 2022-12-28 16:10:03,892 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 29)
[Server:main-server] 2022-12-28 16:10:03,892 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 30)
[Server:main-server] 2022-12-28 16:10:03,892 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 31)
[Server:main-server] 2022-12-28 16:10:03,892 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 32)
[Server:main-server] 2022-12-28 16:10:03,893 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 33)
[Server:main-server] 2022-12-28 16:10:12,278 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.impl.ElasticApmTracerBuilder - Beginning scheduled configuration reload (interval is 30 sec)...
[Server:main-server] 2022-12-28 16:10:12,280 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.impl.ElasticApmTracerBuilder - Finished scheduled configuration reload
[Server:main-server] 2022-12-28 16:10:13,896 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving WAKEUP event (sequence 34)
[Server:main-server] 2022-12-28 16:10:13,896 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Request flush because the request timeout occurred
[Server:main-server] 2022-12-28 16:10:13,896 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Flushing 2257 uncompressed 786 compressed bytes
[Server:main-server] 2022-12-28 16:10:18,901 [elastic-apm-server-reporter] WARN  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - null
[Server:main-server] 2022-12-28 16:10:18,901 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 16 seconds (+/-10%)
[Server:main-server] 2022-12-28 16:10:33,555 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Read timed out, response code is -1
[Server:main-server] 2022-12-28 16:10:33,556 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Sending payload to APM server failed
[Server:main-server]        at co.elastic.apm.agent.report.AbstractIntakeApiHandler.endRequest(AbstractIntakeApiHandler.java:187) [elastic-apm-agent-1.35.0.jar:1.35.0]
[Server:main-server]        at co.elastic.apm.agent.report.AbstractIntakeApiHandler.endRequest(AbstractIntakeApiHandler.java:163) [elastic-apm-agent-1.35.0.jar:1.35.0]
[Server:main-server]        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:85) [elastic-apm-agent-1.35.0.jar:1.35.0]
[Server:main-server]        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:38) [elastic-apm-agent-1.35.0.jar:1.35.0]
[Server:main-server]        at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) [elastic-apm-agent-1.35.0.jar:1.35.0]
[Server:main-server]        at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) [elastic-apm-agent-1.35.0.jar:1.35.0]
[Server:main-server]        at co.elastic.apm.agent.util.ExecutorUtils$2.run(ExecutorUtils.java:99) [elastic-apm-agent-1.35.0.jar:1.35.0]

Any idea?
Thanks

Is your APM Server reachable from your JBoss host? If you run curl http://ELASTICKSERVER:8200 from this host, what do you get?

Is there a proxy/load-balancer in front of your APM Server?

@Eyal_Koren yes, i have curl and telnet from jboss to elastic host.
There is no proxy between them, they locate on same network.

Please take a look in your APM Server logs then, see if you spot something interesting there.
If not, I will need to see the full agent debug log, from startup and until after some traffic had been processed, in order to find out more.

@Eyal_Koren

1-can't attach file here. how can i send it to you?

2-should i use trace annotation in my code? or when i run jboss with apm agent automatically detect traces?

Any idea?
Thanks

You can use gist

It depends on what your application is using. Take a look at our supported technologies - everything that is listed there should be traced automatically without requiring any changes to your code.

@Eyal_Koren

Thanks for sharing.
It still seems like a setup issue to me, but I could be missing something.
My best guess is that the connection is being terminated at the server side before the agent gets the chance to properly flush data and end it.
A common cause for that is the request being timed out at the server side.
If the communication goes directly from the APM agent to the APM Server, this may happen if the APM Server is configured with read_timeout smaller than the request timeout configured for the agent through api_request_time (the default is 10 seconds and it is not modified in your agent).
So, two things to verify:

  • The APM Server read_timeout configuration needs to be set to something >10s (the default is 30s).
  • The communication from the APM Agent goes directly to the APM Server and not through any load-balancer/reverse-proxy. You replied before that "they locate on same network", but this doesn't mean the connection goes directly from one to the other. If there is something in the middle that times out requests, this can be the culprit

@Eyal_Koren
increase read_timeout to 60s but still get same error.

Any other idea?

Did you look in the APM Server logs? Try to set it to debug as well and look for any activity from your Java agent.

@Eyal_Koren same error message on debug log show

[Server:main-server] 2023-01-02 09:35:39,388 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Starting new request to http://ELASTICSERVER:8200/intake/v2/events
[Server:main-server] 2023-01-02 09:35:44,392 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error trying to connect to APM Server at http://ELASTICSERVER:8200/intake/v2/events. Although not necessarily related to SSL, some related SSL configurations corresponding the current connection are logged at INFO level.
[Server:main-server] 2023-01-02 09:35:44,392 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICSET_JSON_WRITER with this error: connect timed out
[Server:main-server] 2023-01-02 09:35:44,392 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Event handling failure
[Server:main-server] java.net.SocketTimeoutException: connect timed out
[Server:main-server]        at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_312]
[Server:main-server]        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_312]
[Server:main-server]        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_312]
[Server:main-server]        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_312]
[Server:main-server]        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_312]
[Server:main-server]        at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_312]
[Server:main-server]        at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[?:1.8.0_312]
[Server:main-server]        at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[?:1.8.0_312]
[Server:main-server]        at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[?:1.8.0_312]
[Server:main-server]        at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) ~[?:1.8.0_312]
[Server:main-server]        at sun.net.www.http.HttpClient.New(HttpClient.java:339) ~[?:1.8.0_312]
[Server:main-server]        at sun.net.www.http.HttpClient.New(HttpClient.java:357) ~[?:1.8.0_312]
[Server:main-server]        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1223) ~[?:1.8.0_312]
[Server:main-server]        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162) ~[?:1.8.0_312]
[Server:main-server]        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056) ~[?:1.8.0_312]
[Server:main-server]        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990) ~[?:1.8.0_312]
[Server:main-server]        at co.elastic.apm.agent.report.AbstractIntakeApiHandler.startRequest(AbstractIntakeApiHandler.java:110) ~[elastic-apm-agent-1.35.0.jar:1.35.0]
[Server:main-server]        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.startRequest(IntakeV2ReportingEventHandler.java:186) [elastic-apm-agent-1.35.0.jar:1.35.0]
[Server:main-server]        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.handleIntakeEvent(IntakeV2ReportingEventHandler.java:132) [elastic-apm-agent-1.35.0.jar:1.35.0]
[Server:main-server]        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.dispatchEvent(IntakeV2ReportingEventHandler.java:117) [elastic-apm-agent-1.35.0.jar:1.35.0]
[Server:main-server]        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:87) [elastic-apm-agent-1.35.0.jar:1.35.0]
[Server:main-server]        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:38) [elastic-apm-agent-1.35.0.jar:1.35.0]
[Server:main-server]        at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) [elastic-apm-agent-1.35.0.jar:1.35.0]
[Server:main-server]        at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) [elastic-apm-agent-1.35.0.jar:1.35.0]
[Server:main-server]        at co.elastic.apm.agent.util.ExecutorUtils$2.run(ExecutorUtils.java:99) [elastic-apm-agent-1.35.0.jar:1.35.0]
[Server:main-server]        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
[Server:main-server] 2023-01-02 09:35:44,395 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)
[Server:main-server] 2023-01-02 09:35:49,502 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.impl.ElasticApmTracerBuilder - Beginning scheduled configuration reload (interval is 30 sec)...
[Server:main-server] 2023-01-02 09:35:49,504 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.impl.ElasticApmTracerBuilder - Finished scheduled configuration reload

This is not an APM Server log, this is still APM Java agent log and not with the same error. In the latter, the agent fails to create a connection with the APM Server. In the former log, there was a connection and the failures occur when ending the requests.
Please avoid opening multiple topics for the same issue.

@Eyal_Koren this is agent log.

APM Agent > APM Server

curl http://ELASTICSERVERIP:8200
{
  "build_date": "2022-12-05T04:10:50Z",
  "build_sha": "397552f2571bc657c9fb93399920331e2cd6bfa3",
  "publish_ready": false,
  "version": "8.5.3"
}

APM Agent > APM Server

curl http://ELASTICSERVERIP:8200/intake/v2/events
{
  "accepted": 0,
  "errors": [
    {
      "message": "only POST requests are supported"
    }
  ]
}

APM Agent > APM Server

telnet ELASTICSERVERIP 8200
Trying ELASTICSERVERIP...
Connected to ELASTICSERVERIP.
Escape character is '^]'.

APM Server

netstat -tulpn  | grep 8200
tcp        0      0 0.0.0.0:8200            0.0.0.0:*               LISTEN      892565/apm-server

any idea?
Thanks

No, no idea, sorry.
Above you provided two parts from APM Agent logs indicating two different issues.

Try to login to your APM Server machine and look for the APM Server log. Once you figure out how to find that, set its level to debug and look for hints there.

I cannot assist further without more info.

@Eyal_Koren
1-you mention "APM Agent logs indicating two different issues" what are these issue?

2-apm agent can't send any data to apm server.
so I try to get tcpdump on apm server machine, no data send from apm agent.

on apm agent I set log level on debug mode.

this is current issue. no data send to the apm server.

The log you shared says:

[Server:main-server] 2022-12-31 13:14:49,310 [elastic-apm-server-healthcheck] INFO  co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: {  "build_date": "2022-12-05T04:10:50Z",  "build_sha": "397552f2571bc657c9fb93399920331e2cd6bfa3",  "publish_ready": false,  "version": "8.5.3"}
[Server:main-server] 2022-12-31 13:14:49,318 [elastic-apm-server-healthcheck] DEBUG co.elastic.apm.agent.report.ApmServerHealthChecker - APM server http://ELASTICKSERVER:8200/ version is: 8.5.3

This means that communication with the APM Server succeeded and the errors were related to ending the requests. I provided the ideas I have for possible reasons for that above. For some reason you won't see what is logged by the APM Server for this communication.

The latest logs you provided indicate another problem - the agent was not able to establish communication with the APM Server.

I am very sorry, but I have no way to assist with this issue,

@Eyal_Koren ok, i try to setup everything from scratch, and now see APM SERVER received traffic from APM AGENT. (I don't have idea what was the issue)

by the way, now enable APM SERVER log in debug mode (as you mention).

here is the flow: (two machine try to send trace to server 1-node app 2-java app)
APM AGENT > (HTTP 8200) APM SERVER > (HTTPS 9200) ELASTIC

as I see in server log:
1-node app: trace accepted by APM SERVER but not write to elastic.

{"log.level":"info","@timestamp":"2023-01-02T18:04:15.566+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (MYAPP)","source.address":"AGENTIP","http.request.id":"9456074d-2054-40b5-80e8-76f3cb8a8bd0","event.duration":10019149314,"http.response.status_code":202,"ecs.version":"1.6.0"}

2-java app: 403,"error.message":"forbidden request

{"log.level":"error","@timestamp":"2023-01-02T18:05:11.839+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"forbidden request","service.name":"apm-server","url.original":"/config/v1/agents","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (MYAPP)","source.address":"AGENTIP","http.request.id":"0f4af159-e48d-4358-926d-834354df0aa9","event.duration":262784,"http.response.status_code":403,"error.message":"forbidden request: Agent remote configuration is disabled. Configure the apm-server.kibanasection in apm-server.yml to enable it. If you are using a RUM agent, you also need to configure theapm-server.rum section. If you are not using remote configuration, you can safely ignore this error.","ecs.version":"1.6.0"}

3- also see certificate issue here

{"log.level":"error","@timestamp":"2023-01-02T18:04:50.885+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}

here is the apm-server.yml

apm-server:
  host: "0.0.0.0:8200"

output.elasticsearch:
  hosts: ["https://localhost:9200"]
  username: "elastic"
  password: "PASS"

  protocol: https
  ssl.enabled: true
  ssl.key: /usr/share/elasticsearch/ca/ca.key
  ssl.certificate: /usr/share/elasticsearch/ca/ca.crt
  ssl.certificate_authorities: /usr/share/elasticsearch/ca/ca.crt

Here is the APM SERVER full log:

{"log.level":"info","@timestamp":"2023-01-02T18:03:50.353+0330","log.origin":{"file.name":"instance/beat.go","file.line":708},"message":"Home path: [/usr/share/apm-server] Config path: [/etc/apm-server] Data path: [/var/lib/apm-server] Logs path: [/var/log/apm-server]","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.353+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":766},"message":"Beat metadata path: /var/lib/apm-server/meta.json","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.353+0330","log.origin":{"file.name":"instance/beat.go","file.line":716},"message":"Beat ID: dd0d058b-81b4-489e-9acd-d1f9b3b1cedc","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.354+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1082},"message":"Beat info","service.name":"apm-server","system_info":{"beat":{"path":{"config":"/etc/apm-server","data":"/var/lib/apm-server","home":"/usr/share/apm-server","logs":"/var/log/apm-server"},"type":"apm-server","uuid":"dd0d058b-81b4-489e-9acd-d1f9b3b1cedc"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.354+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1091},"message":"Build info","service.name":"apm-server","system_info":{"build":{"commit":"397552f2571bc657c9fb93399920331e2cd6bfa3","libbeat":"8.5.3","time":"2022-12-05T04:10:50.000Z","version":"8.5.3"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.354+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1094},"message":"Go runtime info","service.name":"apm-server","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":4,"version":"go1.18.7"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.356+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1098},"message":"Host info","service.name":"apm-server","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-12-21T11:05:20+03:30","containerized":false,"name":"logdev","ip":["127.0.0.1/8","ELASTICIP/21"],"kernel_version":"4.18.0-425.3.1.el8.x86_64","mac":["00:50:56:b7:9c:3f"],"os":{"type":"linux","family":"redhat","platform":"rhel","name":"Red Hat Enterprise Linux","version":"8.7 (Ootpa)","major":8,"minor":7,"patch":0,"codename":"Ootpa"},"timezone":"+0330","timezone_offset_sec":12600,"id":"a578926556b04bf681129e0d2b9db5c6"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.357+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1127},"message":"Process info","service.name":"apm-server","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null},"cwd":"/","exe":"/usr/share/apm-server/bin/apm-server","name":"apm-server","pid":979788,"ppid":1,"seccomp":{"mode":"disabled","no_new_privs":false},"start_time":"2023-01-02T18:03:49.280+0330"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.357+0330","log.origin":{"file.name":"instance/beat.go","file.line":294},"message":"Setup Beat: apm-server; Version: 8.5.3","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.358+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":319},"message":"Initializing output plugins","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-01-02T18:03:50.362+0330","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.363+0330","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":69},"message":"Loading certificate: /usr/share/elasticsearch/ca/ca.crt and key /usr/share/elasticsearch/ca/ca.key","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.363+0330","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":172},"message":"Successfully loaded CA certificate: /usr/share/elasticsearch/ca/ca.crt","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.363+0330","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://localhost:9200","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.364+0330","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":69},"message":"Loading certificate: /usr/share/elasticsearch/ca/ca.crt and key /usr/share/elasticsearch/ca/ca.key","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.364+0330","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":172},"message":"Successfully loaded CA certificate: /usr/share/elasticsearch/ca/ca.crt","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.364+0330","log.logger":"publisher","log.origin":{"file.name":"pipeline/consumer.go","file.line":98},"message":"start pipeline event consumer","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.364+0330","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: logdev","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.364+0330","log.logger":"publisher","log.origin":{"file.name":"pipeline/queue_reader.go","file.line":49},"message":"pipeline event consumer queue reader: start","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.365+0330","log.origin":{"file.name":"instance/beat.go","file.line":471},"message":"apm-server start running.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.366+0330","log.logger":"beater","log.origin":{"file.name":"beater/beater.go","file.line":1162},"message":"maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.374+0330","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":142},"message":"Listening on: 0.0.0.0:8200","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.375+0330","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":69},"message":"Loading certificate: /usr/share/elasticsearch/ca/ca.crt and key /usr/share/elasticsearch/ca/ca.key","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.375+0330","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":172},"message":"Successfully loaded CA certificate: /usr/share/elasticsearch/ca/ca.crt","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.375+0330","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":69},"message":"Loading certificate: /usr/share/elasticsearch/ca/ca.crt and key /usr/share/elasticsearch/ca/ca.key","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.376+0330","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":172},"message":"Successfully loaded CA certificate: /usr/share/elasticsearch/ca/ca.crt","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.376+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":40},"message":"blocking ingestion until all preconditions are satisfied","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.376+0330","log.logger":"beater","log.origin":{"file.name":"apm-server/main.go","file.line":71},"message":"creating transaction metrics aggregation with config: {Interval:1m0s MaxTransactionGroups:10000 HDRHistogramSignificantFigures:2}","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.384+0330","log.logger":"beater","log.origin":{"file.name":"apm-server/main.go","file.line":86},"message":"creating service destinations aggregation with config: {Interval:1m0s MaxGroups:10000}","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.384+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path / added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.384+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /config/v1/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.384+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /config/v1/rum/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.384+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /intake/v2/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.384+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /intake/v3/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.385+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /intake/v2/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.385+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /v1/traces added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.385+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /v1/metrics added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.385+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /v1/logs added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.385+0330","log.logger":"beater","log.origin":{"file.name":"beater/server.go","file.line":193},"message":"Starting apm-server [397552f2571bc657c9fb93399920331e2cd6bfa3 built 2022-12-05 04:10:50 +0000 +0000]. Hit CTRL-C to stop it.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.385+0330","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":94},"message":"RUM endpoints disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.385+0330","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":104},"message":"SSL disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:04:05.513+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:04:15.566+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (MYAPP)","source.address":"AGENTIP","http.request.id":"9456074d-2054-40b5-80e8-76f3cb8a8bd0","event.duration":10019149314,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:04:20.635+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:04:29.330+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.id":"c4f7b9f3-3a56-48cf-a2ed-bba9ce3452f7","event.duration":10012268452,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:04:35.760+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:04:48.423+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.id":"b53fbd5f-f272-45f3-b9e4-91cae1bf64f9","event.duration":10000627328,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:04:50.385+0330","log.logger":"spanmetrics","log.origin":{"file.name":"spanmetrics/aggregator.go","file.line":169},"message":"no span metrics to publish","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:04:50.385+0330","log.logger":"txmetrics","log.origin":{"file.name":"txmetrics/aggregator.go","file.line":215},"message":"no metrics to publish","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:04:50.885+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:04:51.738+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (MYAPP)","source.address":"AGENTIP","http.request.id":"5a599020-3984-475d-b43e-382f963d93ae","event.duration":10004451297,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:05:06.011+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:05:11.839+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"forbidden request","service.name":"apm-server","url.original":"/config/v1/agents","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (MYAPP)","source.address":"AGENTIP","http.request.id":"0f4af159-e48d-4358-926d-834354df0aa9","event.duration":262784,"http.response.status_code":403,"error.message":"forbidden request: Agent remote configuration is disabled. Configure the `apm-server.kibana` section in apm-server.yml to enable it. If you are using a RUM agent, you also need to configure the `apm-server.rum` section. If you are not using remote configuration, you can safely ignore this error.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:05:18.436+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.id":"a6853cc3-df02-4bdd-8ecc-e7b5edba23c1","event.duration":10007274979,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:05:21.149+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:05:21.737+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (MYAPP)","source.address":"AGENTIP","http.request.id":"2a57462a-8558-444d-a44f-e4eb0125b9eb","event.duration":10005458677,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:05:36.274+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:05:48.442+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.id":"4b75b589-fe42-4ca6-9f74-3b2e96e80fe6","event.duration":10009891667,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:05:50.385+0330","log.logger":"txmetrics","log.origin":{"file.name":"txmetrics/aggregator.go","file.line":215},"message":"no metrics to publish","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:05:50.385+0330","log.logger":"spanmetrics","log.origin":{"file.name":"spanmetrics/aggregator.go","file.line":169},"message":"no span metrics to publish","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:05:51.412+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:05:51.734+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (MYAPP)","source.address":"AGENTIP","http.request.id":"04da1f3e-d6e7-4713-93b6-3e7d244aee69","event.duration":10003698267,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:06:06.536+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:06:18.446+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.id":"47ca9ece-ba00-4cfc-99ee-27a37b0dd582","event.duration":10009686961,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:06:21.658+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:06:21.735+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (MYAPP)","source.address":"AGENTIP","http.request.id":"cbbcab63-f53d-4fd1-bda2-4d489fbd6e03","event.duration":10006227001,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:06:36.785+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:06:50.385+0330","log.logger":"txmetrics","log.origin":{"file.name":"txmetrics/aggregator.go","file.line":215},"message":"no metrics to publish","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:06:50.385+0330","log.logger":"spanmetrics","log.origin":{"file.name":"spanmetrics/aggregator.go","file.line":169},"message":"no span metrics to publish","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:06:51.909+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:06:58.455+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"request timed out","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (MYAPP)","source.address":"AGENTIP","http.request.id":"8ef3e9e7-1c78-4bc4-97b3-ecfb8a31471d","event.duration":16719736722,"http.response.status_code":503,"error.message":"request timed out","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:06:58.455+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"request timed out","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.id":"7a45f667-c5da-4d79-b50a-c6c53fcf7027","event.duration":20009864398,"http.response.status_code":503,"error.message":"request timed out","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:07:07.037+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:07:22.164+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:07:28.451+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"request timed out","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.id":"01528a38-cbaf-4dee-a5c3-ab3d3a84845d","event.duration":20004119277,"http.response.status_code":503,"error.message":"request timed out","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:07:28.451+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"request timed out","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (MYAPP)","source.address":"AGENTIP","http.request.id":"fbc38834-9c81-4de4-9b29-e7114ff90e5f","event.duration":16721093029,"http.response.status_code":503,"error.message":"request timed out","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:07:37.289+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:07:37.955+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"forbidden request","service.name":"apm-server","url.original":"/config/v1/agents?service.name=test&service.environment=production","http.request.method":"GET","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.body.bytes":0,"http.request.id":"6263a52b-5d5b-43f4-a15d-7e587df80d59","event.duration":376665,"http.response.status_code":403,"error.message":"forbidden request: Agent remote configuration is disabled. Configure the `apm-server.kibana` section in apm-server.yml to enable it. If you are using a RUM agent, you also need to configure the `apm-server.rum` section. If you are not using remote configuration, you can safely ignore this error.","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:07:50.385+0330","log.logger":"txmetrics","log.origin":{"file.name":"txmetrics/aggregator.go","file.line":215},"message":"no metrics to publish","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:07:50.385+0330","log.logger":"spanmetrics","log.origin":{"file.name":"spanmetrics/aggregator.go","file.line":169},"message":"no span metrics to publish","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:07:52.415+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:07:58.449+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"request timed out","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.id":"fd515408-f430-4a6e-a803-c3b7bcea28ad","event.duration":20003483324,"http.response.status_code":503,"error.message":"request timed out","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:07:58.449+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"request timed out","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (MYAPP)","source.address":"AGENTIP","http.request.id":"3606eef2-605d-481a-b342-d28a2595fb0f","event.duration":16717080431,"http.response.status_code":503,"error.message":"request timed out","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:08:07.555+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}

OK, here is some progress.

No issue here, seems to server event intake from APM Agents properly

This is related to the ability to centrally configure APM Agents through Kibana - when you are using a standalone deployment of the APM Server, there is an additional step required in order to unlock this feature, which is to define a Kibana endpoint within your APM Server configuration. For now, you can ignore this message and only fix that if you want to enable APM Agent central configuration at a later time.

This seems to be a real issue (not the only one) that currently prevents you from using the APM Solution.
Please read our SSL/TLS guide carefully. Specifically, take a look at the Client configuration options section and more specifically, if you are using a self-signed certificate - notice this part:

If certificate_authorities is self-signed, the host system needs to trust that CA cert as well

In addition, both your Java and NodeJS agents fail to properly send events, with these errors in the APM Server log:

{"log.level":"error","@timestamp":"2023-01-02T18:06:58.455+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"request timed out","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (MYAPP)","source.address":"AGENTIP","http.request.id":"8ef3e9e7-1c78-4bc4-97b3-ecfb8a31471d","event.duration":16719736722,"http.response.status_code":503,"error.message":"request timed out","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:06:58.455+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"request timed out","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.id":"7a45f667-c5da-4d79-b50a-c6c53fcf7027","event.duration":20009864398,"http.response.status_code":503,"error.message":"request timed out","ecs.version":"1.6.0"}

Looking in our troubleshooting page, this is coupled with too high load, but it is possible that in this case this is only a symptom of the APM Server not able to send data to Elasticsearch (due to the issue above), and small load is enough to trigger this issue.

I hope this helps.

@Eyal_Koren
ok I disable elastic ssl, now everything are http!

FYI1: here is the apm-server.yml

apm-server:
  host: "0.0.0.0:8200"

logging.level: debug
logging.files:
  path: /var/log/apm-server


apm-server.kibana.enabled: true
apm-server.kibana.host: "http://localhost:5601"
apm-server.kibana.protocol: "http"
apm-server.kibana.username: "elastic"
apm-server.kibana.password: "PASS"

FYI2: Kibana endpoint configuration defined.

here is the AGENT SERVER log:

{"log.level":"info","@timestamp":"2023-01-04T09:34:07.406+0330","log.origin":{"file.name":"instance/beat.go","file.line":708},"message":"Home path: [/usr/share/apm-server] Config path: [/etc/apm-server] Data path: [/var/lib/apm-server] Logs path: [/var/log/apm-server]","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-04T09:34:07.406+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":766},"message":"Beat metadata path: /var/lib/apm-server/meta.json","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.406+0330","log.origin":{"file.name":"instance/beat.go","file.line":716},"message":"Beat ID: dd0d058b-81b4-489e-9acd-d1f9b3b1cedc","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.406+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1082},"message":"Beat info","service.name":"apm-server","system_info":{"beat":{"path":{"config":"/etc/apm-server","data":"/var/lib/apm-server","home":"/usr/share/apm-server","logs":"/var/log/apm-server"},"type":"apm-server","uuid":"dd0d058b-81b4-489e-9acd-d1f9b3b1cedc"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.406+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1091},"message":"Build info","service.name":"apm-server","system_info":{"build":{"commit":"397552f2571bc657c9fb93399920331e2cd6bfa3","libbeat":"8.5.3","time":"2022-12-05T04:10:50.000Z","version":"8.5.3"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.407+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1094},"message":"Go runtime info","service.name":"apm-server","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":4,"version":"go1.18.7"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.408+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1098},"message":"Host info","service.name":"apm-server","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-12-21T11:05:20+03:30","containerized":false,"name":"logdev","ip":["127.0.0.1/8","ELASTICSERVERIP/21"],"kernel_version":"4.18.0-425.3.1.el8.x86_64","mac":["00:50:56:b7:9c:3f"],"os":{"type":"linux","family":"redhat","platform":"rhel","name":"Red Hat Enterprise Linux","version":"8.7 (Ootpa)","major":8,"minor":7,"patch":0,"codename":"Ootpa"},"timezone":"+0330","timezone_offset_sec":12600,"id":"a578926556b04bf681129e0d2b9db5c6"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.409+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1127},"message":"Process info","service.name":"apm-server","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null},"cwd":"/","exe":"/usr/share/apm-server/bin/apm-server","name":"apm-server","pid":1358010,"ppid":1,"seccomp":{"mode":"disabled","no_new_privs":false},"start_time":"2023-01-04T09:34:06.340+0330"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.409+0330","log.origin":{"file.name":"instance/beat.go","file.line":294},"message":"Setup Beat: apm-server; Version: 8.5.3","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-04T09:34:07.410+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":319},"message":"Initializing output plugins","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.415+0330","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: http://localhost:9200","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-04T09:34:07.416+0330","log.logger":"publisher","log.origin":{"file.name":"pipeline/consumer.go","file.line":98},"message":"start pipeline event consumer","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.416+0330","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: logdev","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-04T09:34:07.416+0330","log.logger":"publisher","log.origin":{"file.name":"pipeline/queue_reader.go","file.line":49},"message":"pipeline event consumer queue reader: start","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.417+0330","log.origin":{"file.name":"instance/beat.go","file.line":471},"message":"apm-server start running.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.418+0330","log.logger":"beater","log.origin":{"file.name":"beater/beater.go","file.line":1162},"message":"maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.426+0330","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":142},"message":"Listening on: 0.0.0.0:8200","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-04T09:34:07.426+0330","log.logger":"kibana","log.origin":{"file.name":"kibana/connecting_client.go","file.line":74},"message":"Trying to obtain connection to Kibana.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.426+0330","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":179},"message":"Kibana url: http://localhost:5601","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.427+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":40},"message":"blocking ingestion until all preconditions are satisfied","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.427+0330","log.logger":"agentcfg","log.origin":{"file.name":"agentcfg/cache.go","file.line":38},"message":"Cache creation with expiration 30s.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.427+0330","log.logger":"beater","log.origin":{"file.name":"apm-server/main.go","file.line":71},"message":"creating transaction metrics aggregation with config: {Interval:1m0s MaxTransactionGroups:10000 HDRHistogramSignificantFigures:2}","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.435+0330","log.logger":"beater","log.origin":{"file.name":"apm-server/main.go","file.line":86},"message":"creating service destinations aggregation with config: {Interval:1m0s MaxGroups:10000}","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.436+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path / added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.436+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /config/v1/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.436+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /config/v1/rum/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.436+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /intake/v2/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.436+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /intake/v3/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.436+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /intake/v2/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.436+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /v1/traces added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.436+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /v1/metrics added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.436+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /v1/logs added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.436+0330","log.logger":"beater","log.origin":{"file.name":"beater/server.go","file.line":193},"message":"Starting apm-server [397552f2571bc657c9fb93399920331e2cd6bfa3 built 2022-12-05 04:10:50 +0000 +0000]. Hit CTRL-C to stop it.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.437+0330","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":94},"message":"RUM endpoints disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.437+0330","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":104},"message":"SSL disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:07.448+0330","log.logger":"kibana","log.origin":{"file.name":"kibana/connecting_client.go","file.line":81},"message":"Successfully obtained connection to Kibana.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-04T09:34:22.444+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: EOF","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-04T09:34:28.119+0330","log.logger":"agentcfg","log.origin":{"file.name":"agentcfg/cache.go","file.line":58},"message":"Cache size 1. Added ID testproduction.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:28.119+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"not modified","service.name":"apm-server","url.original":"/config/v1/agents?service.name=test&service.environment=production","http.request.method":"GET","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.body.bytes":0,"http.request.id":"ae7067d6-08cf-44f6-80e6-825f6c53d57c","event.duration":52201091,"http.response.status_code":304,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:37.195+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.id":"047cdd43-3514-497e-87ff-f49c1f74c36c","event.duration":10006019917,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-04T09:34:37.454+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: EOF","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-04T09:34:52.466+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: EOF","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-04T09:34:58.177+0330","log.logger":"agentcfg","log.origin":{"file.name":"agentcfg/cache.go","file.line":58},"message":"Cache size 1. Added ID testproduction.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:34:58.177+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"not modified","service.name":"apm-server","url.original":"/config/v1/agents?service.name=test&service.environment=production","http.request.method":"GET","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.body.bytes":0,"http.request.id":"29089db8-0d95-4892-927c-448cbe1c16cd","event.duration":48003483,"http.response.status_code":304,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:35:07.192+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.id":"216e9780-2437-4673-a7b7-f12c3bbfb3ce","event.duration":10000050247,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-04T09:35:07.436+0330","log.logger":"spanmetrics","log.origin":{"file.name":"spanmetrics/aggregator.go","file.line":169},"message":"no span metrics to publish","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-04T09:35:07.436+0330","log.logger":"txmetrics","log.origin":{"file.name":"txmetrics/aggregator.go","file.line":215},"message":"no metrics to publish","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-04T09:35:07.477+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: EOF","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-04T09:35:22.487+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: EOF","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-04T09:35:28.243+0330","log.logger":"agentcfg","log.origin":{"file.name":"agentcfg/cache.go","file.line":58},"message":"Cache size 1. Added ID testproduction.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:35:28.243+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"not modified","service.name":"apm-server","url.original":"/config/v1/agents?service.name=test&service.environment=production","http.request.method":"GET","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.body.bytes":0,"http.request.id":"76670a9f-5a21-4899-bff7-26c77a473613","event.duration":50353193,"http.response.status_code":304,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-04T09:35:37.202+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.id":"0b2ff84d-eeee-4c35-98f8-2243146825a2","event.duration":10005106387,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-04T09:35:37.499+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: EOF","service.name":"apm-server","ecs.version":"1.6.0"}

Hi @Indeed2000 ,
the server cannot retrieve whether the APM integration has been installed, which is a precondition before ingesting data. The APM integration takes care of setting up index templates, ILM policies, etc. Please follow how to install APM integration if you haven't already.

The apm-server.yml only shows the configuration to Kibana, previously there were TLS errors when trying to connect to Elasticsearch. Can you please also share the configuration to Elasticsearch, and ensure that both, Kibana and Elasticsearch are reachable from where your APM Server is running.