Configure app logging in APM

Elastic Observability APM
,
1

Kibana version: 8.0.0

Elasticsearch version: 8.0.0

APM Server version: 8.0.0

APM Agent language and version: Java & 1.29.0

Browser version: Chrome

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Downloaded

Fresh install or upgraded from other version? Fresh

Is there anything special in your setup? No

Steps to reproduce:
I have deployed apm server, with ES and KB on Azure.
I have Java app where i configured apm agent.
On APM app, I'm not able to see logs.
How can i transfer app logs

image
image1885×779 79.7 KB

image
image945×271 15.2 KB

2

The java app configuration, at startup should tell you if you have connected to the APM Server... are you verifying this?

If you have no data... there is an error logged somewhere or a misconfiguration. How is your java app configured with apm agent ?

3

Yeah, from java app logs, it says to be connected

exec java -XX:MinRAMPercentage=70 -XX:MaxRAMPercentage=70 -javaagent:/app/apm-agent.jar -Delastic.apm.server_url=https://10.0.0.0:8200 -Delastic.apm.verify_server_cert=false -Delastic.apm.service_name=apm-rules-data -Delastic.apm.application_packages=com.test --add-opens java.base/java.math=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.time=ALL-UNNAMED -jar rules.jar
[main] INFO  co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.sun.management.OperatingSystemMXBean
[main] INFO  co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.29.0 as apm-rules-data (0.0.1-SNAPSHOT) on Java 17.0.2 Runtime version: 17.0.2+8-86 VM version: 17.0.2+8-86 (Oracle Corporation) Linux 5.4.0-1069-azure
[main] INFO  co.elastic.apm.agent.configuration.StartupInfo - service_name: 'apm-rules-data' (source: Java System Properties)
[main] INFO  co.elastic.apm.agent.configuration.StartupInfo - server_url: 'https://10.0.0.0:8200' (source: Java System Properties)
[main] INFO  co.elastic.apm.agent.configuration.StartupInfo - verify_server_cert: 'false' (source: Java System Properties)
[main] INFO  co.elastic.apm.agent.configuration.StartupInfo - application_packages: 'com.test' (source: Java System Properties)
[main] INFO  co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
[elastic-apm-server-healthcheck] INFO  co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: {  "build_date": "2022-02-03T16:50:25Z",  "build_sha": "6afb387ae002e5a8dc2427b53225aa42681e4208",  "publish_ready": true,  "version": "8.0.0"}
[elastic-apm-remote-config-poller] INFO  co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Received new configuration from APM Server: {}

APP is configured manually with apm agent

JAVA_OPTS="${JAVA_OPTS} -javaagent:/app/apm-agent.jar"
JAVA_OPTS="${JAVA_OPTS} -Delastic.apm.server_url=${APM_SERVER_URI}"
JAVA_OPTS="${JAVA_OPTS} -Delastic.apm.verify_server_cert=false"
JAVA_OPTS="${JAVA_OPTS} -Delastic.apm.service_name=apm-rules-data"
JAVA_OPTS="${JAVA_OPTS} -Delastic.apm.application_packages=com.test"

At least it should show server start up logs, right?

4

The AMM agent is registered correctly with the Service Name so that is good. it seems more that there are no transactions or spans to send ? This is more to do with your application, you can turn on the debug for more details on if your java application is sending anything.

5

HealthCheck is basic transaction which is logging at INFO level and can be seen in app log.
Same transaction and traces is present in APM.

image
image1874×531 73.7 KB

logs

2022-04-09 14:02:34,524 http-nio-9001-exec-6 INFO   com.test.rules.controller.RulesController: Health check ok
2022-04-09 14:02:41,553 http-nio-9001-exec-7 INFO   com.test.rules.controller.RulesController: Health check ok
2022-04-09 14:03:04,524 http-nio-9001-exec-8 INFO   com.test.rules.controller.RulesController: Health check ok
2022-04-09 14:03:11,553 http-nio-9001-exec-9 INFO   com.test.rules.controller.RulesController: Health check ok

Is there any other configuration need to be done on APM to poll the logs? eg: Fleet agent?

I think, logging integration is required to embedd logs into the APM UI. But how :thinking: :thinking:

6

If something is being registered, then the APM agent seems fine, maybe this is your application issue ?

7

What i understood from logs is, we need to app logs in ES to correlate with APM

So i used fluent bit to send logs to ES

Now, how can we correlate fluent bit logs to APM logs?

8

Normally this will be via the timestamp, but for proper instrumentation, there should be a correlation id present in both the APM & Logs.

9

Please see i have transaction id in metadata of APM transaction and In Kibana we have logs for this transaction id, still it does n't show in APM logs

image
image1676×765 71 KB

image
image1885×558 72.8 KB

Streams are not present and logs are not correlating
image
image1889×734 76.1 KB

10

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.