@Eyal_Koren ok, i try to setup everything from scratch, and now see APM SERVER received traffic from APM AGENT. (I don't have idea what was the issue)
by the way, now enable APM SERVER log in debug mode (as you mention).
here is the flow: (two machine try to send trace to server 1-node app 2-java app)
APM AGENT > (HTTP 8200) APM SERVER > (HTTPS 9200) ELASTIC
as I see in server log:
1-node app: trace accepted by APM SERVER but not write to elastic.
{"log.level":"info","@timestamp":"2023-01-02T18:04:15.566+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (MYAPP)","source.address":"AGENTIP","http.request.id":"9456074d-2054-40b5-80e8-76f3cb8a8bd0","event.duration":10019149314,"http.response.status_code":202,"ecs.version":"1.6.0"}
2-java app: 403,"error.message":"forbidden request
{"log.level":"error","@timestamp":"2023-01-02T18:05:11.839+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"forbidden request","service.name":"apm-server","url.original":"/config/v1/agents","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (MYAPP)","source.address":"AGENTIP","http.request.id":"0f4af159-e48d-4358-926d-834354df0aa9","event.duration":262784,"http.response.status_code":403,"error.message":"forbidden request: Agent remote configuration is disabled. Configure the apm-server.kibanasection in apm-server.yml to enable it. If you are using a RUM agent, you also need to configure theapm-server.rum section. If you are not using remote configuration, you can safely ignore this error.","ecs.version":"1.6.0"}
3- also see certificate issue here
{"log.level":"error","@timestamp":"2023-01-02T18:04:50.885+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
here is the apm-server.yml
apm-server:
host: "0.0.0.0:8200"
output.elasticsearch:
hosts: ["https://localhost:9200"]
username: "elastic"
password: "PASS"
protocol: https
ssl.enabled: true
ssl.key: /usr/share/elasticsearch/ca/ca.key
ssl.certificate: /usr/share/elasticsearch/ca/ca.crt
ssl.certificate_authorities: /usr/share/elasticsearch/ca/ca.crt
Here is the APM SERVER full log:
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.353+0330","log.origin":{"file.name":"instance/beat.go","file.line":708},"message":"Home path: [/usr/share/apm-server] Config path: [/etc/apm-server] Data path: [/var/lib/apm-server] Logs path: [/var/log/apm-server]","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.353+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":766},"message":"Beat metadata path: /var/lib/apm-server/meta.json","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.353+0330","log.origin":{"file.name":"instance/beat.go","file.line":716},"message":"Beat ID: dd0d058b-81b4-489e-9acd-d1f9b3b1cedc","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.354+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1082},"message":"Beat info","service.name":"apm-server","system_info":{"beat":{"path":{"config":"/etc/apm-server","data":"/var/lib/apm-server","home":"/usr/share/apm-server","logs":"/var/log/apm-server"},"type":"apm-server","uuid":"dd0d058b-81b4-489e-9acd-d1f9b3b1cedc"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.354+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1091},"message":"Build info","service.name":"apm-server","system_info":{"build":{"commit":"397552f2571bc657c9fb93399920331e2cd6bfa3","libbeat":"8.5.3","time":"2022-12-05T04:10:50.000Z","version":"8.5.3"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.354+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1094},"message":"Go runtime info","service.name":"apm-server","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":4,"version":"go1.18.7"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.356+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1098},"message":"Host info","service.name":"apm-server","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-12-21T11:05:20+03:30","containerized":false,"name":"logdev","ip":["127.0.0.1/8","ELASTICIP/21"],"kernel_version":"4.18.0-425.3.1.el8.x86_64","mac":["00:50:56:b7:9c:3f"],"os":{"type":"linux","family":"redhat","platform":"rhel","name":"Red Hat Enterprise Linux","version":"8.7 (Ootpa)","major":8,"minor":7,"patch":0,"codename":"Ootpa"},"timezone":"+0330","timezone_offset_sec":12600,"id":"a578926556b04bf681129e0d2b9db5c6"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.357+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1127},"message":"Process info","service.name":"apm-server","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null},"cwd":"/","exe":"/usr/share/apm-server/bin/apm-server","name":"apm-server","pid":979788,"ppid":1,"seccomp":{"mode":"disabled","no_new_privs":false},"start_time":"2023-01-02T18:03:49.280+0330"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.357+0330","log.origin":{"file.name":"instance/beat.go","file.line":294},"message":"Setup Beat: apm-server; Version: 8.5.3","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.358+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":319},"message":"Initializing output plugins","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-01-02T18:03:50.362+0330","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.363+0330","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":69},"message":"Loading certificate: /usr/share/elasticsearch/ca/ca.crt and key /usr/share/elasticsearch/ca/ca.key","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.363+0330","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":172},"message":"Successfully loaded CA certificate: /usr/share/elasticsearch/ca/ca.crt","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.363+0330","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://localhost:9200","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.364+0330","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":69},"message":"Loading certificate: /usr/share/elasticsearch/ca/ca.crt and key /usr/share/elasticsearch/ca/ca.key","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.364+0330","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":172},"message":"Successfully loaded CA certificate: /usr/share/elasticsearch/ca/ca.crt","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.364+0330","log.logger":"publisher","log.origin":{"file.name":"pipeline/consumer.go","file.line":98},"message":"start pipeline event consumer","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.364+0330","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: logdev","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.364+0330","log.logger":"publisher","log.origin":{"file.name":"pipeline/queue_reader.go","file.line":49},"message":"pipeline event consumer queue reader: start","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.365+0330","log.origin":{"file.name":"instance/beat.go","file.line":471},"message":"apm-server start running.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.366+0330","log.logger":"beater","log.origin":{"file.name":"beater/beater.go","file.line":1162},"message":"maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.374+0330","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":142},"message":"Listening on: 0.0.0.0:8200","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.375+0330","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":69},"message":"Loading certificate: /usr/share/elasticsearch/ca/ca.crt and key /usr/share/elasticsearch/ca/ca.key","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.375+0330","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":172},"message":"Successfully loaded CA certificate: /usr/share/elasticsearch/ca/ca.crt","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.375+0330","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":69},"message":"Loading certificate: /usr/share/elasticsearch/ca/ca.crt and key /usr/share/elasticsearch/ca/ca.key","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:03:50.376+0330","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":172},"message":"Successfully loaded CA certificate: /usr/share/elasticsearch/ca/ca.crt","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.376+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":40},"message":"blocking ingestion until all preconditions are satisfied","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.376+0330","log.logger":"beater","log.origin":{"file.name":"apm-server/main.go","file.line":71},"message":"creating transaction metrics aggregation with config: {Interval:1m0s MaxTransactionGroups:10000 HDRHistogramSignificantFigures:2}","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.384+0330","log.logger":"beater","log.origin":{"file.name":"apm-server/main.go","file.line":86},"message":"creating service destinations aggregation with config: {Interval:1m0s MaxGroups:10000}","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.384+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path / added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.384+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /config/v1/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.384+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /config/v1/rum/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.384+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /intake/v2/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.384+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /intake/v3/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.385+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /intake/v2/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.385+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /v1/traces added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.385+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /v1/metrics added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.385+0330","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /v1/logs added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.385+0330","log.logger":"beater","log.origin":{"file.name":"beater/server.go","file.line":193},"message":"Starting apm-server [397552f2571bc657c9fb93399920331e2cd6bfa3 built 2022-12-05 04:10:50 +0000 +0000]. Hit CTRL-C to stop it.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.385+0330","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":94},"message":"RUM endpoints disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:03:50.385+0330","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":104},"message":"SSL disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:04:05.513+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:04:15.566+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (MYAPP)","source.address":"AGENTIP","http.request.id":"9456074d-2054-40b5-80e8-76f3cb8a8bd0","event.duration":10019149314,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:04:20.635+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:04:29.330+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.id":"c4f7b9f3-3a56-48cf-a2ed-bba9ce3452f7","event.duration":10012268452,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:04:35.760+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:04:48.423+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.id":"b53fbd5f-f272-45f3-b9e4-91cae1bf64f9","event.duration":10000627328,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:04:50.385+0330","log.logger":"spanmetrics","log.origin":{"file.name":"spanmetrics/aggregator.go","file.line":169},"message":"no span metrics to publish","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:04:50.385+0330","log.logger":"txmetrics","log.origin":{"file.name":"txmetrics/aggregator.go","file.line":215},"message":"no metrics to publish","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:04:50.885+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:04:51.738+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (MYAPP)","source.address":"AGENTIP","http.request.id":"5a599020-3984-475d-b43e-382f963d93ae","event.duration":10004451297,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:05:06.011+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:05:11.839+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"forbidden request","service.name":"apm-server","url.original":"/config/v1/agents","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (MYAPP)","source.address":"AGENTIP","http.request.id":"0f4af159-e48d-4358-926d-834354df0aa9","event.duration":262784,"http.response.status_code":403,"error.message":"forbidden request: Agent remote configuration is disabled. Configure the `apm-server.kibana` section in apm-server.yml to enable it. If you are using a RUM agent, you also need to configure the `apm-server.rum` section. If you are not using remote configuration, you can safely ignore this error.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:05:18.436+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.id":"a6853cc3-df02-4bdd-8ecc-e7b5edba23c1","event.duration":10007274979,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:05:21.149+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:05:21.737+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (MYAPP)","source.address":"AGENTIP","http.request.id":"2a57462a-8558-444d-a44f-e4eb0125b9eb","event.duration":10005458677,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:05:36.274+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:05:48.442+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.id":"4b75b589-fe42-4ca6-9f74-3b2e96e80fe6","event.duration":10009891667,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:05:50.385+0330","log.logger":"txmetrics","log.origin":{"file.name":"txmetrics/aggregator.go","file.line":215},"message":"no metrics to publish","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:05:50.385+0330","log.logger":"spanmetrics","log.origin":{"file.name":"spanmetrics/aggregator.go","file.line":169},"message":"no span metrics to publish","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:05:51.412+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:05:51.734+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (MYAPP)","source.address":"AGENTIP","http.request.id":"04da1f3e-d6e7-4713-93b6-3e7d244aee69","event.duration":10003698267,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:06:06.536+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:06:18.446+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.id":"47ca9ece-ba00-4cfc-99ee-27a37b0dd582","event.duration":10009686961,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:06:21.658+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-02T18:06:21.735+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (MYAPP)","source.address":"AGENTIP","http.request.id":"cbbcab63-f53d-4fd1-bda2-4d489fbd6e03","event.duration":10006227001,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:06:36.785+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:06:50.385+0330","log.logger":"txmetrics","log.origin":{"file.name":"txmetrics/aggregator.go","file.line":215},"message":"no metrics to publish","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:06:50.385+0330","log.logger":"spanmetrics","log.origin":{"file.name":"spanmetrics/aggregator.go","file.line":169},"message":"no span metrics to publish","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:06:51.909+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:06:58.455+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"request timed out","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (MYAPP)","source.address":"AGENTIP","http.request.id":"8ef3e9e7-1c78-4bc4-97b3-ecfb8a31471d","event.duration":16719736722,"http.response.status_code":503,"error.message":"request timed out","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:06:58.455+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"request timed out","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.id":"7a45f667-c5da-4d79-b50a-c6c53fcf7027","event.duration":20009864398,"http.response.status_code":503,"error.message":"request timed out","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:07:07.037+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:07:22.164+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:07:28.451+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"request timed out","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.id":"01528a38-cbaf-4dee-a5c3-ab3d3a84845d","event.duration":20004119277,"http.response.status_code":503,"error.message":"request timed out","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:07:28.451+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"request timed out","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (MYAPP)","source.address":"AGENTIP","http.request.id":"fbc38834-9c81-4de4-9b29-e7114ff90e5f","event.duration":16721093029,"http.response.status_code":503,"error.message":"request timed out","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:07:37.289+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:07:37.955+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"forbidden request","service.name":"apm-server","url.original":"/config/v1/agents?service.name=test&service.environment=production","http.request.method":"GET","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.body.bytes":0,"http.request.id":"6263a52b-5d5b-43f4-a15d-7e587df80d59","event.duration":376665,"http.response.status_code":403,"error.message":"forbidden request: Agent remote configuration is disabled. Configure the `apm-server.kibana` section in apm-server.yml to enable it. If you are using a RUM agent, you also need to configure the `apm-server.rum` section. If you are not using remote configuration, you can safely ignore this error.","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:07:50.385+0330","log.logger":"txmetrics","log.origin":{"file.name":"txmetrics/aggregator.go","file.line":215},"message":"no metrics to publish","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2023-01-02T18:07:50.385+0330","log.logger":"spanmetrics","log.origin":{"file.name":"spanmetrics/aggregator.go","file.line":169},"message":"no span metrics to publish","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:07:52.415+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:07:58.449+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"request timed out","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/3.41.1 (test 1.0.0)","source.address":"NODCLIENTIP","http.request.id":"fd515408-f430-4a6e-a803-c3b7bcea28ad","event.duration":20003483324,"http.response.status_code":503,"error.message":"request timed out","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:07:58.449+0330","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"request timed out","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (MYAPP)","source.address":"AGENTIP","http.request.id":"3606eef2-605d-481a-b342-d28a2595fb0f","event.duration":16717080431,"http.response.status_code":503,"error.message":"request timed out","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-02T18:08:07.555+0330","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: x509: certificate signed by unknown authority","service.name":"apm-server","ecs.version":"1.6.0"}