Configure app logging in APM

[Raushan]

Kibana version: 8.0.0

Elasticsearch version: 8.0.0

APM Server version: 8.0.0

APM Agent language and version: Java & 1.29.0

Browser version: Chrome

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Downloaded

Fresh install or upgraded from other version? Fresh

Is there anything special in your setup? No

Steps to reproduce:
I have deployed apm server, with ES and KB on Azure.
I have Java app where i configured apm agent.
On APM app, I'm not able to see logs.
How can i transfer app logs

[zx8086]

The java app configuration, at startup should tell you if you have connected to the APM Server... are you verifying this?

If you have no data... there is an error logged somewhere or a misconfiguration. How is your java app configured with apm agent ?

[Raushan]

Yeah, from java app logs, it says to be connected

exec java -XX:MinRAMPercentage=70 -XX:MaxRAMPercentage=70 -javaagent:/app/apm-agent.jar -Delastic.apm.server_url=https://10.0.0.0:8200 -Delastic.apm.verify_server_cert=false -Delastic.apm.service_name=apm-rules-data -Delastic.apm.application_packages=com.test --add-opens java.base/java.math=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.time=ALL-UNNAMED -jar rules.jar
[main] INFO  co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.sun.management.OperatingSystemMXBean
[main] INFO  co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.29.0 as apm-rules-data (0.0.1-SNAPSHOT) on Java 17.0.2 Runtime version: 17.0.2+8-86 VM version: 17.0.2+8-86 (Oracle Corporation) Linux 5.4.0-1069-azure
[main] INFO  co.elastic.apm.agent.configuration.StartupInfo - service_name: 'apm-rules-data' (source: Java System Properties)
[main] INFO  co.elastic.apm.agent.configuration.StartupInfo - server_url: 'https://10.0.0.0:8200' (source: Java System Properties)
[main] INFO  co.elastic.apm.agent.configuration.StartupInfo - verify_server_cert: 'false' (source: Java System Properties)
[main] INFO  co.elastic.apm.agent.configuration.StartupInfo - application_packages: 'com.test' (source: Java System Properties)
[main] INFO  co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
[elastic-apm-server-healthcheck] INFO  co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: {  "build_date": "2022-02-03T16:50:25Z",  "build_sha": "6afb387ae002e5a8dc2427b53225aa42681e4208",  "publish_ready": true,  "version": "8.0.0"}
[elastic-apm-remote-config-poller] INFO  co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Received new configuration from APM Server: {}

APP is configured manually with apm agent

JAVA_OPTS="${JAVA_OPTS} -javaagent:/app/apm-agent.jar"
JAVA_OPTS="${JAVA_OPTS} -Delastic.apm.server_url=${APM_SERVER_URI}"
JAVA_OPTS="${JAVA_OPTS} -Delastic.apm.verify_server_cert=false"
JAVA_OPTS="${JAVA_OPTS} -Delastic.apm.service_name=apm-rules-data"
JAVA_OPTS="${JAVA_OPTS} -Delastic.apm.application_packages=com.test"

At least it should show server start up logs, right?

[zx8086]

The AMM agent is registered correctly with the Service Name so that is good. it seems more that there are no transactions or spans to send ? This is more to do with your application, you can turn on the debug for more details on if your java application is sending anything.

[Raushan]

HealthCheck is basic transaction which is logging at INFO level and can be seen in app log.
Same transaction and traces is present in APM.

logs

2022-04-09 14:02:34,524 http-nio-9001-exec-6 INFO   com.test.rules.controller.RulesController: Health check ok
2022-04-09 14:02:41,553 http-nio-9001-exec-7 INFO   com.test.rules.controller.RulesController: Health check ok
2022-04-09 14:03:04,524 http-nio-9001-exec-8 INFO   com.test.rules.controller.RulesController: Health check ok
2022-04-09 14:03:11,553 http-nio-9001-exec-9 INFO   com.test.rules.controller.RulesController: Health check ok

Is there any other configuration need to be done on APM to poll the logs? eg: Fleet agent?

I think, logging integration is required to embedd logs into the APM UI. But how

[zx8086]

If something is being registered, then the APM agent seems fine, maybe this is your application issue ?

Log correlation | APM Java Agent Reference [1.x] | Elastic

API/Code | APM Java Agent Reference [1.x] | Elastic

[Raushan]

@zx8086

hen the APM agent seems fine, maybe this is your application issue ?

What i understood from logs is, we need to app logs in ES to correlate with APM

So i used fluent bit to send logs to ES

Now, how can we correlate fluent bit logs to APM logs?

[zx8086]

Normally this will be via the timestamp, but for proper instrumentation, there should be a correlation id present in both the APM & Logs.

[Raushan]

Please see i have transaction id in metadata of APM transaction and In Kibana we have logs for this transaction id, still it does n't show in APM logs

Streams are not present and logs are not correlating