I've followed [Programmatic API setup to self-attach | APM Java Agent Reference [1.x] | Elastic](https://Programmatic setup to self-attach) and I was able to see everything being tracked except the Logs tab under Trace sample. I always see "There are no log messages to display" every time I try to refresh. I don't know what I'm missing here. I was expecting the application logs would be pulled here. Please help me if there's anything I didn't get right.
logs are not automatically sent. There are a variety of ways to have logs sent to the APM server, but I suggest you start with log_sending which if it works for you is the easiest
1 Like
I added log_sending=true and checked it. unfortunately, it didn't work for me.
Can you check the agent debug output for errors?
I have the same issue.
this is my test controller
@RestController
@RequestMapping("/log")
public class LogController {
private static Logger logger = LoggerFactory.getLogger(LogController.class);
@Autowired
ObjectMapper mapper;
@PostMapping(consumes = "application/json")
public ResponseEntity<?> log(@RequestBody Map<?,?> body) throws JsonProcessingException {
logger.info("123");
logger.warn("456");
logger.error("789");
return ResponseEntity.ok(body);
}
}
and here is agent properties
-Delastic.apm.service_name=<service name>
-Delastic.apm.server_urls=http://localhost:8200
-Delastic.apm.secret_token=
-Delastic.apm.environment=dev
-Delastic.apm.log_sending=true
-Delastic.apm.application_packages=<app package>
-Delastic.apm.log_ecs_reformatting=OVERRIDE
-Delastic.apm.enable_log_correlation=true
and here is the syslog when trigger the api
{"@timestamp":"2024-09-04T08:38:11.620Z","log.level": "INFO","message":"123","ecs.version": "1.2.0","service.name":"playground","service.environment":"dev","event.dataset":"playground.CONSOLE","process.thread.name":"http-nio-8080-exec-4","log.logger":"<package>","transaction.id":"6c2082a1f6306bf0","trace.id":"ac99aa13f72f67f39eba851b663f4ff1"}
{"@timestamp":"2024-09-04T08:38:11.620Z","log.level": "WARN","message":"456","ecs.version": "1.2.0","service.name":"playground","service.environment":"dev","event.dataset":"playground.CONSOLE","process.thread.name":"http-nio-8080-exec-4","log.logger":"<package>","transaction.id":"6c2082a1f6306bf0","trace.id":"ac99aa13f72f67f39eba851b663f4ff1"}
{"@timestamp":"2024-09-04T08:38:11.620Z","log.level":"ERROR","message":"789","ecs.version": "1.2.0","service.name":"playground","service.environment":"dev","event.dataset":"playground.CONSOLE","process.thread.name":"http-nio-8080-exec-4","log.logger":"<package>","transaction.id":"6c2082a1f6306bf0","trace.id":"ac99aa13f72f67f39eba851b663f4ff1"}
Which version of the apm Java agent are you using, and have you set the agent logs to debug and looked for errors?
I am using elastic-apm-agent-1.51.0.jar
after add -Delastic.apm.log_level=debug, here's the printout
2024-09-05 08:57:29,250 [http-nio-8080-exec-3] DEBUG co.elastic.apm.agent.impl.transaction.AbstractRefCountedContext - increment references to '' 00-fcedc01be69a2896e363579650512a8e-dea267c473f2547a-01 (4639255c) (1)
2024-09-05 08:57:29,250 [http-nio-8080-exec-3] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - startTransaction '' 00-fcedc01be69a2896e363579650512a8e-dea267c473f2547a-01 (4639255c)
2024-09-05 08:57:29,250 [http-nio-8080-exec-3] DEBUG co.elastic.apm.agent.impl.ActiveStack - Activating '' 00-fcedc01be69a2896e363579650512a8e-dea267c473f2547a-01 (4639255c) on thread 43
2024-09-05 08:57:29,250 [http-nio-8080-exec-3] DEBUG co.elastic.apm.agent.impl.transaction.AbstractRefCountedContext - increment references to '' 00-fcedc01be69a2896e363579650512a8e-dea267c473f2547a-01 (4639255c) (2)
2024-09-05 08:57:29,251 [http-nio-8080-exec-3] DEBUG co.elastic.apm.agent.servlet.ServletTransactionHelper - Not capturing Request body because the capture_body config option is OFF
2024-09-05 08:57:29,251 [http-nio-8080-exec-3] DEBUG co.elastic.apm.agent.servlet.ServletTransactionHelper - Not capturing body for content type "application/json". Consider updating the capture_body_content_types configuration option.
{"@timestamp":"2024-09-05T00:57:29.252Z","log.level": "INFO","message":"123","ecs.version": "1.2.0","service.name":"playground","service.environment":"dev","event.dataset":"playground.CONSOLE","process.thread.name":"http-nio-8080-exec-3","log.logger":"vulcan.springboot.playground.controller.LogController","transaction.id":"dea267c473f2547a","trace.id":"fcedc01be69a2896e363579650512a8e"}
{"@timestamp":"2024-09-05T00:57:29.253Z","log.level": "WARN","message":"456","ecs.version": "1.2.0","service.name":"playground","service.environment":"dev","event.dataset":"playground.CONSOLE","process.thread.name":"http-nio-8080-exec-3","log.logger":"vulcan.springboot.playground.controller.LogController","transaction.id":"dea267c473f2547a","trace.id":"fcedc01be69a2896e363579650512a8e"}
{"@timestamp":"2024-09-05T00:57:29.253Z","log.level":"ERROR","message":"789","ecs.version": "1.2.0","service.name":"playground","service.environment":"dev","event.dataset":"playground.CONSOLE","process.thread.name":"http-nio-8080-exec-3","log.logger":"vulcan.springboot.playground.controller.LogController","transaction.id":"dea267c473f2547a","trace.id":"fcedc01be69a2896e363579650512a8e"}
2024-09-05 08:57:29,254 [http-nio-8080-exec-3] DEBUG co.elastic.apm.agent.impl.ActiveStack - Deactivating 'LogController#log' 00-fcedc01be69a2896e363579650512a8e-dea267c473f2547a-01 (4639255c) on thread 43
2024-09-05 08:57:29,254 [http-nio-8080-exec-3] DEBUG co.elastic.apm.agent.impl.transaction.AbstractRefCountedContext - decrement references to 'LogController#log' 00-fcedc01be69a2896e363579650512a8e-dea267c473f2547a-01 (4639255c) (1)
2024-09-05 08:57:29,254 [http-nio-8080-exec-3] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - endTransaction 'LogController#log' 00-fcedc01be69a2896e363579650512a8e-dea267c473f2547a-01 (4639255c)
2024-09-05 08:57:29,255 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 656)
2024-09-05 08:57:29,255 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.util.UrlConnectionUtils - Opening http://localhost:8200/intake/v2/events without proxy
2024-09-05 08:57:29,255 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.AbstractIntakeApiHandler - Starting new request to http://localhost:8200/intake/v2/events
2024-09-05 08:57:29,257 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Scheduling request timeout in 10 seconds
2024-09-05 08:57:29,257 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 657)
2024-09-05 08:57:29,257 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 658)
2024-09-05 08:57:29,257 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 659)
2024-09-05 08:57:29,257 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 660)
2024-09-05 08:57:29,257 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 661)
2024-09-05 08:57:29,257 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 662)
2024-09-05 08:57:29,257 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 663)
2024-09-05 08:57:29,257 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 664)
2024-09-05 08:57:29,257 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 665)
2024-09-05 08:57:29,257 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 666)
2024-09-05 08:57:29,257 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 667)
2024-09-05 08:57:29,257 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving TRANSACTION event (sequence 668)
2024-09-05 08:57:29,257 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractRefCountedContext - decrement references to 'LogController#log' 00-fcedc01be69a2896e363579650512a8e-dea267c473f2547a-01 (4639255c) (0)
2024-09-05 08:57:29,257 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 669)
2024-09-05 08:57:29,257 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 670)
2024-09-05 08:57:35,011 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.impl.ElasticApmTracerBuilder - Beginning scheduled configuration reload (interval is 30 sec)...
2024-09-05 08:57:35,011 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 671)
2024-09-05 08:57:35,011 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.impl.ElasticApmTracerBuilder - Finished scheduled configuration reload
2024-09-05 08:57:35,011 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 672)
2024-09-05 08:57:35,563 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 673)
2024-09-05 08:57:35,563 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 674)
2024-09-05 08:57:35,563 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 675)
2024-09-05 08:57:35,563 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 676)
2024-09-05 08:57:35,563 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 677)
2024-09-05 08:57:35,563 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 678)
2024-09-05 08:57:35,563 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 679)
2024-09-05 08:57:35,563 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 680)
2024-09-05 08:57:35,563 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 681)
2024-09-05 08:57:35,563 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 682)
2024-09-05 08:57:39,273 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving WAKEUP event (sequence 683)
2024-09-05 08:57:39,273 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Request flush because the request timeout occurred
2024-09-05 08:57:39,274 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.AbstractIntakeApiHandler - Flushing 0 uncompressed 4221 compressed bytes
2024-09-05 08:58:05,015 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.impl.ElasticApmTracerBuilder - Beginning scheduled configuration reload (interval is 30 sec)...
2024-09-05 08:58:05,015 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.impl.ElasticApmTracerBuilder - Finished scheduled configuration reload
2024-09-05 08:58:05,016 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 684)
2024-09-05 08:58:05,017 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.util.UrlConnectionUtils - Opening http://localhost:8200/intake/v2/events without proxy
2024-09-05 08:58:05,017 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.AbstractIntakeApiHandler - Starting new request to http://localhost:8200/intake/v2/events
2024-09-05 08:58:05,019 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Scheduling request timeout in 10 seconds
2024-09-05 08:58:05,019 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 685)
2024-09-05 08:58:05,019 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 686)
2024-09-05 08:58:05,564 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 687)
2024-09-05 08:58:05,564 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 688)
2024-09-05 08:58:05,564 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 689)
2024-09-05 08:58:05,564 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 690)
2024-09-05 08:58:05,564 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 691)
2024-09-05 08:58:05,564 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 692)
2024-09-05 08:58:05,564 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 693)
2024-09-05 08:58:05,565 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 694)
2024-09-05 08:58:05,565 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 695)
bump that to trace ( -Delastic.apm.log_level=trace) which shows the actual JSON being sent, and look for any log that is being sent (should be lines starting with {"log":
-Delastic.apm.log_level=trace
result
2024-09-07 17:46:10,037 [elastic-apm-circuit-breaker] TRACE co.elastic.apm.agent.impl.circuitbreaker.CircuitBreaker - Scheduling next stress monitor polling in 5000s
2024-09-07 17:46:10,041 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1097)
2024-09-07 17:46:10,041 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Request flush because the request timeout occurred
2024-09-07 17:46:10,041 [elastic-apm-server-reporter] TRACE co.elastic.apm.agent.report.serialize.DslJsonSerializer - {"metadata":{"service":{"name":"elastic-apm-agent","environment":"dev","agent":{"name":"java","ephemeral_id":"0c24c58d-f1cd-4132-8c49-fd5d511bb6c8","version":"1.51.0"},"language":{"name":"Java","version":"17.0.12"},"runtime":{"name":"Java","version":"17.0.12"},"version":null},"process":{"pid":3076,"ppid":2895,"title":"/Library/Java/JavaVirtualMachines/zulu-17.jdk/Contents/Home/bin/java"},"system":{"architecture":"aarch64","detected_hostname":"vulcansmbp.local","platform":"Mac OS X"}}}
2024-09-07 17:46:10,042 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.AbstractIntakeApiHandler - Flushing 0 uncompressed 490 compressed bytes
2024-09-07 17:46:10,044 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.util.UrlConnectionUtils - Opening http://localhost:8200/intake/v2/events without proxy
2024-09-07 17:46:10,044 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.AbstractIntakeApiHandler - Starting new request to http://localhost:8200/intake/v2/events
2024-09-07 17:46:10,046 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Scheduling request timeout in 10 seconds
2024-09-07 17:46:10,046 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving WAKEUP event (sequence 1098)
2024-09-07 17:46:10,046 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1099)
2024-09-07 17:46:11,787 [http-nio-8080-exec-4] TRACE co.elastic.apm.agent.httpserver.HttpServerHelper - No matcher found for excluding this request with path: /log and User-Agent: PostmanRuntime/7.41.2
2024-09-07 17:46:11,787 [http-nio-8080-exec-4] DEBUG co.elastic.apm.agent.impl.transaction.AbstractRefCountedContext - increment references to '' 00-70f666afdc72331bc1bf67f62a335d8d-0966fdef7d99e52d-01 (19c1748f) (1)
2024-09-07 17:46:11,787 [http-nio-8080-exec-4] TRACE co.elastic.apm.agent.impl.transaction.AbstractRefCountedContext - incrementing references at
java.lang.RuntimeException: This is an expected exception. Is just used to record where the reference count has been incremented.
at co.elastic.apm.agent.impl.transaction.AbstractRefCountedContext.incrementReferences(AbstractRefCountedContext.java:43) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.transaction.TransactionImpl.incrementReferences(TransactionImpl.java:400) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.transaction.AbstractSpanImpl.onAfterStart(AbstractSpanImpl.java:541) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.transaction.TransactionImpl.onTransactionStart(TransactionImpl.java:187) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.transaction.TransactionImpl.start(TransactionImpl.java:172) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.ElasticApmTracer.startChildTransaction(ElasticApmTracer.java:356) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.ElasticApmTracer.startChildTransaction(ElasticApmTracer.java:348) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.ElasticApmTracer.startChildTransaction(ElasticApmTracer.java:323) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.ElasticApmTracer.startChildTransaction(ElasticApmTracer.java:102) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.tracer.GlobalTracer.startChildTransaction(GlobalTracer.java:134) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.servlet.ServletTransactionHelper.createAndActivateTransaction(ServletTransactionHelper.java:86) [elastic-apm-agent-1.51.0.jar:?]
at co.elastic.apm.agent.servlet.ServletApiAdvice.onServletEnter(ServletApiAdvice.java:102) [elastic-apm-agent-1.51.0.jar:?]
at co.elastic.apm.agent.servlet.JakartaServletApiAdvice.onEnterServletService(JakartaServletApiAdvice.java:35) [elastic-apm-agent-1.51.0.jar:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:119) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:384) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:904) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) [tomcat-embed-core-10.1.28.jar:10.1.28]
at java.lang.Thread.run(Thread.java:840) [?:?]
2024-09-07 17:46:11,788 [http-nio-8080-exec-4] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - startTransaction '' 00-70f666afdc72331bc1bf67f62a335d8d-0966fdef7d99e52d-01 (19c1748f)
2024-09-07 17:46:11,788 [http-nio-8080-exec-4] TRACE co.elastic.apm.agent.impl.ElasticApmTracer - starting transaction at
java.lang.RuntimeException: this exception is just used to record where the transaction has been started from
at co.elastic.apm.agent.impl.ElasticApmTracer.afterTransactionStart(ElasticApmTracer.java:367) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.ElasticApmTracer.startChildTransaction(ElasticApmTracer.java:358) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.ElasticApmTracer.startChildTransaction(ElasticApmTracer.java:348) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.ElasticApmTracer.startChildTransaction(ElasticApmTracer.java:323) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.ElasticApmTracer.startChildTransaction(ElasticApmTracer.java:102) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.tracer.GlobalTracer.startChildTransaction(GlobalTracer.java:134) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.servlet.ServletTransactionHelper.createAndActivateTransaction(ServletTransactionHelper.java:86) [elastic-apm-agent-1.51.0.jar:?]
at co.elastic.apm.agent.servlet.ServletApiAdvice.onServletEnter(ServletApiAdvice.java:102) [elastic-apm-agent-1.51.0.jar:?]
at co.elastic.apm.agent.servlet.JakartaServletApiAdvice.onEnterServletService(JakartaServletApiAdvice.java:35) [elastic-apm-agent-1.51.0.jar:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:119) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:384) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:904) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) [tomcat-embed-core-10.1.28.jar:10.1.28]
at java.lang.Thread.run(Thread.java:840) [?:?]
2024-09-07 17:46:11,788 [http-nio-8080-exec-4] DEBUG co.elastic.apm.agent.impl.ActiveStack - Activating '' 00-70f666afdc72331bc1bf67f62a335d8d-0966fdef7d99e52d-01 (19c1748f) on thread 44
2024-09-07 17:46:11,788 [http-nio-8080-exec-4] DEBUG co.elastic.apm.agent.impl.transaction.AbstractRefCountedContext - increment references to '' 00-70f666afdc72331bc1bf67f62a335d8d-0966fdef7d99e52d-01 (19c1748f) (2)
2024-09-07 17:46:11,788 [http-nio-8080-exec-4] TRACE co.elastic.apm.agent.impl.transaction.AbstractRefCountedContext - incrementing references at
java.lang.RuntimeException: This is an expected exception. Is just used to record where the reference count has been incremented.
at co.elastic.apm.agent.impl.transaction.AbstractRefCountedContext.incrementReferences(AbstractRefCountedContext.java:43) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.transaction.TransactionImpl.incrementReferences(TransactionImpl.java:400) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.ActiveStack.activate(ActiveStack.java:98) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.ElasticApmTracer.activate(ElasticApmTracer.java:917) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.transaction.TraceStateImpl.activate(TraceStateImpl.java:64) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.transaction.TransactionImpl.activate(TransactionImpl.java:207) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.transaction.TransactionImpl.activate(TransactionImpl.java:51) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.servlet.ServletTransactionHelper.createAndActivateTransaction(ServletTransactionHelper.java:88) [elastic-apm-agent-1.51.0.jar:?]
at co.elastic.apm.agent.servlet.ServletApiAdvice.onServletEnter(ServletApiAdvice.java:102) [elastic-apm-agent-1.51.0.jar:?]
at co.elastic.apm.agent.servlet.JakartaServletApiAdvice.onEnterServletService(JakartaServletApiAdvice.java:35) [elastic-apm-agent-1.51.0.jar:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:119) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:384) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:904) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) [tomcat-embed-core-10.1.28.jar:10.1.28]
at java.lang.Thread.run(Thread.java:840) [?:?]
2024-09-07 17:46:11,789 [http-nio-8080-exec-4] DEBUG co.elastic.apm.agent.servlet.ServletTransactionHelper - Not capturing Request body because the capture_body config option is OFF
2024-09-07 17:46:11,789 [http-nio-8080-exec-4] DEBUG co.elastic.apm.agent.servlet.ServletTransactionHelper - Not capturing body for content type "application/json". Consider updating the capture_body_content_types configuration option.
{"@timestamp":"2024-09-07T09:46:11.791Z","log.level": "INFO","message":"123","ecs.version": "1.2.0","service.name":"elastic-apm-agent","service.environment":"dev","event.dataset":"elastic-apm-agent.CONSOLE","process.thread.name":"http-nio-8080-exec-4","log.logger":"vulcan.springboot.elastic_apm_agent.controller.LogController","transaction.id":"0966fdef7d99e52d","trace.id":"70f666afdc72331bc1bf67f62a335d8d"}
{"@timestamp":"2024-09-07T09:46:11.791Z","log.level": "WARN","message":"456","ecs.version": "1.2.0","service.name":"elastic-apm-agent","service.environment":"dev","event.dataset":"elastic-apm-agent.CONSOLE","process.thread.name":"http-nio-8080-exec-4","log.logger":"vulcan.springboot.elastic_apm_agent.controller.LogController","transaction.id":"0966fdef7d99e52d","trace.id":"70f666afdc72331bc1bf67f62a335d8d"}
{"@timestamp":"2024-09-07T09:46:11.791Z","log.level":"ERROR","message":"789","ecs.version": "1.2.0","service.name":"elastic-apm-agent","service.environment":"dev","event.dataset":"elastic-apm-agent.CONSOLE","process.thread.name":"http-nio-8080-exec-4","log.logger":"vulcan.springboot.elastic_apm_agent.controller.LogController","transaction.id":"0966fdef7d99e52d","trace.id":"70f666afdc72331bc1bf67f62a335d8d"}
2024-09-07 17:46:11,792 [http-nio-8080-exec-4] DEBUG co.elastic.apm.agent.impl.ActiveStack - Deactivating 'LogController#log' 00-70f666afdc72331bc1bf67f62a335d8d-0966fdef7d99e52d-01 (19c1748f) on thread 44
2024-09-07 17:46:11,792 [http-nio-8080-exec-4] DEBUG co.elastic.apm.agent.impl.transaction.AbstractRefCountedContext - decrement references to 'LogController#log' 00-70f666afdc72331bc1bf67f62a335d8d-0966fdef7d99e52d-01 (19c1748f) (1)
2024-09-07 17:46:11,792 [http-nio-8080-exec-4] TRACE co.elastic.apm.agent.impl.transaction.AbstractRefCountedContext - decrementing references at
java.lang.RuntimeException: This is an expected exception. Is just used to record where the reference count has been decremented.
at co.elastic.apm.agent.impl.transaction.AbstractRefCountedContext.decrementReferences(AbstractRefCountedContext.java:55) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.ActiveStack.deactivate(ActiveStack.java:129) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.ElasticApmTracer.deactivate(ElasticApmTracer.java:940) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.transaction.TraceStateImpl.deactivate(TraceStateImpl.java:71) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.transaction.TraceStateImpl.deactivate(TraceStateImpl.java:33) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.servlet.ServletTransactionHelper.onAfter(ServletTransactionHelper.java:246) [elastic-apm-agent-1.51.0.jar:?]
at co.elastic.apm.agent.servlet.ServletApiAdvice.onExitServlet(ServletApiAdvice.java:277) [elastic-apm-agent-1.51.0.jar:?]
at co.elastic.apm.agent.servlet.JakartaServletApiAdvice.onExitServletService(JakartaServletApiAdvice.java:45) [elastic-apm-agent-1.51.0.jar:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:142) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:384) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:904) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) [tomcat-embed-core-10.1.28.jar:10.1.28]
at java.lang.Thread.run(Thread.java:840) [?:?]
2024-09-07 17:46:11,793 [http-nio-8080-exec-4] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - endTransaction 'LogController#log' 00-70f666afdc72331bc1bf67f62a335d8d-0966fdef7d99e52d-01 (19c1748f)
2024-09-07 17:46:11,793 [http-nio-8080-exec-4] TRACE co.elastic.apm.agent.impl.ElasticApmTracer - ending transaction at
java.lang.RuntimeException: this exception is just used to record where the transaction has been ended from
at co.elastic.apm.agent.impl.ElasticApmTracer.endTransaction(ElasticApmTracer.java:550) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.transaction.TransactionImpl.afterEnd(TransactionImpl.java:292) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.transaction.AbstractSpanImpl.end(AbstractSpanImpl.java:579) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.impl.transaction.AbstractSpanImpl.end(AbstractSpanImpl.java:549) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.servlet.ServletTransactionHelper.onAfter(ServletTransactionHelper.java:248) [elastic-apm-agent-1.51.0.jar:?]
at co.elastic.apm.agent.servlet.ServletApiAdvice.onExitServlet(ServletApiAdvice.java:277) [elastic-apm-agent-1.51.0.jar:?]
at co.elastic.apm.agent.servlet.JakartaServletApiAdvice.onExitServletService(JakartaServletApiAdvice.java:45) [elastic-apm-agent-1.51.0.jar:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:142) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:384) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:904) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-embed-core-10.1.28.jar:10.1.28]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) [tomcat-embed-core-10.1.28.jar:10.1.28]
at java.lang.Thread.run(Thread.java:840) [?:?]
2024-09-07 17:46:11,795 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1100)
2024-09-07 17:46:11,795 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1101)
2024-09-07 17:46:11,795 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1102)
2024-09-07 17:46:11,795 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1103)
2024-09-07 17:46:11,795 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1104)
2024-09-07 17:46:11,795 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1105)
2024-09-07 17:46:11,795 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1106)
2024-09-07 17:46:11,795 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1107)
2024-09-07 17:46:11,795 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1108)
2024-09-07 17:46:11,795 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1109)
2024-09-07 17:46:11,795 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1110)
2024-09-07 17:46:11,795 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1111)
2024-09-07 17:46:11,795 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1112)
2024-09-07 17:46:11,795 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1113)
2024-09-07 17:46:11,795 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1114)
2024-09-07 17:46:11,795 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1115)
2024-09-07 17:46:11,795 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1116)
2024-09-07 17:46:11,795 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1117)
2024-09-07 17:46:11,795 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving TRANSACTION event (sequence 1118)
2024-09-07 17:46:11,796 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractRefCountedContext - decrement references to 'LogController#log' 00-70f666afdc72331bc1bf67f62a335d8d-0966fdef7d99e52d-01 (19c1748f) (0)
2024-09-07 17:46:11,796 [elastic-apm-server-reporter] TRACE co.elastic.apm.agent.impl.transaction.AbstractRefCountedContext - decrementing references at
java.lang.RuntimeException: This is an expected exception. Is just used to record where the reference count has been decremented.
at co.elastic.apm.agent.impl.transaction.AbstractRefCountedContext.decrementReferences(AbstractRefCountedContext.java:55) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.report.ReportingEvent.end(ReportingEvent.java:172) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:96) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:39) [elastic-apm-agent-1.51.0.jar:1.51.0]
at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168) [elastic-apm-agent-1.51.0.jar:1.51.0]
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125) [elastic-apm-agent-1.51.0.jar:1.51.0]
at co.elastic.apm.agent.util.ExecutorUtils$2.run(ExecutorUtils.java:99) [elastic-apm-agent-1.51.0.jar:1.51.0]
at java.lang.Thread.run(Thread.java:840) [?:?]
2024-09-07 17:46:11,796 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1119)
2024-09-07 17:46:11,796 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1120)
2024-09-07 17:46:15,039 [elastic-apm-circuit-breaker] TRACE co.elastic.apm.agent.impl.circuitbreaker.CircuitBreaker - Scheduling next stress monitor polling in 5000s
2024-09-07 17:46:15,047 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1121)
2024-09-07 17:46:19,182 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.impl.ElasticApmTracerBuilder - Beginning scheduled configuration reload (interval is 30 sec)...
2024-09-07 17:46:19,182 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.impl.ElasticApmTracerBuilder - Finished scheduled configuration reload
2024-09-07 17:46:19,186 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1122)
2024-09-07 17:46:19,186 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1123)
2024-09-07 17:46:19,956 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 1124)
2024-09-07 17:46:19,956 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 1125)
2024-09-07 17:46:19,956 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 1126)
2024-09-07 17:46:19,956 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 1127)
2024-09-07 17:46:19,956 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 1128)
2024-09-07 17:46:19,956 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 1129)
2024-09-07 17:46:19,956 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 1130)
2024-09-07 17:46:19,956 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 1131)
2024-09-07 17:46:19,956 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 1132)
2024-09-07 17:46:19,956 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICSET_JSON_WRITER event (sequence 1133)
2024-09-07 17:46:20,041 [elastic-apm-circuit-breaker] TRACE co.elastic.apm.agent.impl.circuitbreaker.CircuitBreaker - Scheduling next stress monitor polling in 5000s
2024-09-07 17:46:20,045 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1134)
2024-09-07 17:46:20,053 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving WAKEUP event (sequence 1135)
2024-09-07 17:46:20,053 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Request flush because the request timeout occurred
2024-09-07 17:46:20,053 [elastic-apm-server-reporter] TRACE co.elastic.apm.agent.report.serialize.DslJsonSerializer - {"metadata":{"service":{"name":"elastic-apm-agent","environment":"dev","agent":{"name":"java","ephemeral_id":"0c24c58d-f1cd-4132-8c49-fd5d511bb6c8","version":"1.51.0"},"language":{"name":"Java","version":"17.0.12"},"runtime":{"name":"Java","version":"17.0.12"},"version":null},"process":{"pid":3076,"ppid":2895,"title":"/Library/Java/JavaVirtualMachines/zulu-17.jdk/Contents/Home/bin/java"},"system":{"architecture":"aarch64","detected_hostname":"vulcansmbp.local","platform":"Mac OS X"}}}
{"transaction":{"timestamp":1725702371787013,"name":"LogController#log","id":"0966fdef7d99e52d","trace_id":"70f666afdc72331bc1bf67f62a335d8d","type":"request","duration":5.247,"result":"HTTP 2xx","outcome":"success","context":{"service":{"framework":{"version":"6.1.12","name":"Spring Web MVC"},"version":null},"request":{"method":"POST","headers":{"content-type":"application/json","user-agent":"PostmanRuntime/7.41.2","accept":"*/*","cache-control":"no-cache","postman-token":"[REDACTED]","host":"localhost:8080","accept-encoding":"gzip, deflate, br","connection":"keep-alive","content-length":"26"},"body":"[REDACTED]","url":{"full":"http://localhost:8080/log","hostname":"localhost","port":8080,"pathname":"/log","protocol":"http"},"socket":{"remote_address":"0:0:0:0:0:0:0:1"},"http_version":"1.1"},"response":{"headers":{"Content-Type":"application/json","Transfer-Encoding":"chunked","Date":"Sat, 07 Sep 2024 09:46:11 GMT","Keep-Alive":"timeout=60","Connection":"keep-alive"},"finished":true,"headers_sent":true,"status_code":200},"tags":{}},"span_count":{"dropped":0,"started":0},"dropped_spans_stats":[],"sample_rate":1.0,"sampled":true}}
{"metricset":{"timestamp":1725702379951000,"samples":{"system.process.cpu.total.norm.pct":{"value":0.0004701961300702577},"jvm.memory.heap.used":{"value":87915760.0},"jvm.memory.non_heap.used":{"value":100500920.0},"jvm.memory.heap.max":{"value":9663676416.0},"jvm.fd.used":{"value":72.0},"jvm.gc.alloc":{"value":40695080.0},"jvm.memory.non_heap.committed":{"value":102170624.0},"system.cpu.total.norm.pct":{"value":0.11297300591066967},"system.process.memory.size":{"value":432772841472.0},"jvm.memory.heap.committed":{"value":201326592.0},"system.memory.actual.free":{"value":229818368.0},"jvm.fd.max":{"value":2147483647.0},"system.memory.total":{"value":38654705664.0},"jvm.thread.count":{"value":31.0}}}}
{"metricset":{"timestamp":1725702379951000,"tags":{"name":"Metaspace"},"samples":{"jvm.memory.non_heap.pool.committed":{"value":77725696.0},"jvm.memory.non_heap.pool.used":{"value":76961232.0}}}}
{"metricset":{"timestamp":1725702379951000,"tags":{"name":"G1 Survivor Space"},"samples":{"jvm.memory.heap.pool.committed":{"value":8388608.0},"jvm.memory.heap.pool.used":{"value":2560752.0}}}}
{"metricset":{"timestamp":1725702379951000,"tags":{"name":"G1 Old Gen"},"samples":{"jvm.memory.heap.pool.committed":{"value":117440512.0},"jvm.memory.heap.pool.max":{"value":9663676416.0},"jvm.memory.heap.pool.used":{"value":60189184.0}}}}
{"metricset":{"timestamp":1725702379951000,"tags":{"name":"CodeCache"},"samples":{"jvm.memory.non_heap.pool.committed":{"value":12582912.0},"jvm.memory.non_heap.pool.used":{"value":12005888.0},"jvm.memory.non_heap.pool.max":{"value":50331648.0}}}}
{"metricset":{"timestamp":1725702379951000,"tags":{"name":"G1 Young Generation"},"samples":{"jvm.gc.time":{"value":52.0},"jvm.gc.count":{"value":15.0}}}}
{"metricset":{"timestamp":1725702379951000,"tags":{"name":"G1 Old Generation"},"samples":{"jvm.gc.time":{"value":0.0},"jvm.gc.count":{"value":0.0}}}}
{"metricset":{"timestamp":1725702379951000,"tags":{"name":"G1 Eden Space"},"samples":{"jvm.memory.heap.pool.committed":{"value":75497472.0},"jvm.memory.heap.pool.used":{"value":25165824.0}}}}
{"metricset":{"timestamp":1725702379951000,"transaction":{"name":"LogController#log","type":"request"},"span":{"type":"app","subtype":null},"tags":{},"samples":{"span.self_time.count":{"value":1},"span.self_time.sum.us":{"value":5247}}}}
{"metricset":{"timestamp":1725702379951000,"tags":{"name":"Compressed Class Space"},"samples":{"jvm.memory.non_heap.pool.committed":{"value":11862016.0},"jvm.memory.non_heap.pool.used":{"value":11533800.0},"jvm.memory.non_heap.pool.max":{"value":1073741824.0}}}}
2024-09-07 17:46:20,053 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.AbstractIntakeApiHandler - Flushing 0 uncompressed 4223 compressed bytes
2024-09-07 17:46:25,046 [elastic-apm-circuit-breaker] TRACE co.elastic.apm.agent.impl.circuitbreaker.CircuitBreaker - Scheduling next stress monitor polling in 5000s
2024-09-07 17:46:25,058 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1136)
2024-09-07 17:46:25,058 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.util.UrlConnectionUtils - Opening http://localhost:8200/intake/v2/events without proxy
2024-09-07 17:46:25,058 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.AbstractIntakeApiHandler - Starting new request to http://localhost:8200/intake/v2/events
2024-09-07 17:46:25,061 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Scheduling request timeout in 10 seconds
2024-09-07 17:46:25,061 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1137)
2024-09-07 17:46:30,052 [elastic-apm-circuit-breaker] TRACE co.elastic.apm.agent.impl.circuitbreaker.CircuitBreaker - Scheduling next stress monitor polling in 5000s
2024-09-07 17:46:30,055 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1138)
2024-09-07 17:46:35,052 [elastic-apm-circuit-breaker] TRACE co.elastic.apm.agent.impl.circuitbreaker.CircuitBreaker - Scheduling next stress monitor polling in 5000s
2024-09-07 17:46:35,053 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1139)
2024-09-07 17:46:35,068 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving WAKEUP event (sequence 1140)
2024-09-07 17:46:35,069 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Request flush because the request timeout occurred
2024-09-07 17:46:35,069 [elastic-apm-server-reporter] TRACE co.elastic.apm.agent.report.serialize.DslJsonSerializer - {"metadata":{"service":{"name":"elastic-apm-agent","environment":"dev","agent":{"name":"java","ephemeral_id":"0c24c58d-f1cd-4132-8c49-fd5d511bb6c8","version":"1.51.0"},"language":{"name":"Java","version":"17.0.12"},"runtime":{"name":"Java","version":"17.0.12"},"version":null},"process":{"pid":3076,"ppid":2895,"title":"/Library/Java/JavaVirtualMachines/zulu-17.jdk/Contents/Home/bin/java"},"system":{"architecture":"aarch64","detected_hostname":"vulcansmbp.local","platform":"Mac OS X"}}}
2024-09-07 17:46:35,069 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.AbstractIntakeApiHandler - Flushing 0 uncompressed 490 compressed bytes
2024-09-07 17:46:40,054 [elastic-apm-circuit-breaker] TRACE co.elastic.apm.agent.impl.circuitbreaker.CircuitBreaker - Scheduling next stress monitor polling in 5000s
2024-09-07 17:46:40,055 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1141)
2024-09-07 17:46:40,055 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.util.UrlConnectionUtils - Opening http://localhost:8200/intake/v2/events without proxy
2024-09-07 17:46:40,055 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.AbstractIntakeApiHandler - Starting new request to http://localhost:8200/intake/v2/events
2024-09-07 17:46:40,056 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Scheduling request timeout in 10 seconds
2024-09-07 17:46:40,056 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving BYTES_LOG event (sequence 1142)
Okay that shows it's not even trying to send any logs. So that means it's not instrumenting the logging framework. I'm not a spring person, but I think the default logging framework for Spring is commons logging - which we don't instrument. I think most spring deployments use one of the main logging frameworks (log4j2, logback, or j.u.l) and these are what we instrument.
1 Like
according to these docs
https://docs.spring.io/spring-boot/how-to/logging.html
https://docs.spring.io/spring-boot/reference/features/logging.html
springboot default logging framework is logback
in my application I use slf4j to do the log action
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
private static final Logger logger = LoggerFactory.getLogger(LogController.class);
I will try log4j2 and j.u.l
It supports Logback by default but doesn't enable it. That first link says
To use Logback, you need to include it and
spring-jclon the classpath
Thank you, I'll keep on trying.
I'm using spring-boot-starter-web as project dependency, so it should be enabled by default
After multiple attempts, I still cannot see the application log on APM. I am certain I have used the Logback framework, and here is the configuration of my logback.xml.
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<include resource="org/springframework/boot/logging/logback/base.xml"/>
<logger name="org.springframework.web" level="TRACE"/>
</configuration>
the log code
var logger = org.sl4j.LoggerFactory.getLogger(LogController.class);
logger.error("@@@@@LOG@@@@@@@");
elasticstack and apm-server version are 7.17.24
elastic-java-agent version is 1.51.0
with apm config
elastic.apm.log_sending=true
elastic.apm.application_packages=<pacakge>
elastic.apm.log_ecs_reformatting=override
elastic.apm.enable_log_correlation=true
elastic.apm.log_level=trace
There is still an issue with APM where it fails to display application logs, despite successfully rendering traces and metrics.
