How to capture the Logj4 logs created in java service to APM Server

Elastic, Kibana, APM version : 7.7.0

APM java agent version - 1.16.0:

Chrome latest:

Created SAAS elastic cloud acct:

**Fresh install **

**Is there anything special in your setup?**NO.

I have created a java service and also i have linked my java service to elastic APM using elastic APM agent

  <dependency>
  <groupId>co.elastic.apm</groupId>
  <artifactId>apm-agent-attach</artifactId>
  <version>1.16.0</version>
  </dependency>

As described in document attached apm attached to my main method.

ElasticApmAttacher.attach();

Created elasticapm.properties file in my java service

elasticapm.properties

    service_name=my-cool-service
    application_packages=com.psic.ddddd
    server_urls=https://dddddddddddddddddd.apm.eastus2.azure.elastic-cloud.com
    secret_token=dddddd
    log_level=INFO
    enable_log_correlation=true
    log_file=_AGENT_HOME_/logs/elastic-apm.log
    log_format_file=JSON
    log_format_sout=JSON

when ever i run the service i am able to get all the system and infra related logs in my APM server but my actual intention is to get the logs written in the java service using logger factory. Logger printing all the logs to my eclipse console but there is no sign of my custom created logs in APM or even the elastic-apm.log file which i have created.

JAVA SERVICE

@GetMapping("fetch")
@Produces(value = "application/json")
@ApiOperation("Get user detail ")
public FinalResponse getUserDetails(@RequestParam String agentId) {
    LOGGER.info("Decrypted auth token : " + authTokendecrypted);

	LOGGER.info("Incoming agentId : " + agentId);

	try {

	LOGGER.info("Response Success");

	} catch (MyRestTemplateException execption) {

		LOGGER.error(execption.getMessage());


	} catch (Exception ex) {

		LOGGER.error(ex.getMessage());

	} finally {

		LOGGER.debug("we are in finally block");

	}

	LOGGER.info("Get the user details information from OKTA Service End");

	return finalResponse;

}

My java service will be going to write the logs using logger like

import org.slf4j.LoggerFactory;
import ch.qos.logback.classic.Logger;

Unfortunately i am unable to find these logs in APM server and also the log file which i have created using apm agent log_file=AGENT_HOME/logs/elastic-apm.log.

Could some one help me to resolve the issue. Also let me know how to ship the custom created logs of my service to APM server. It would be really helpful for me to analyze and monitor all the application, system and custom created logs in APM server itself.

elastic-apm log file for reference

2020-06-06 12:45:48.701 [Attach Listener] INFO co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.sun.management.OperatingSystemMXBean
2020-06-06 12:45:48.763 [Attach Listener] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.16.0 as my-cool-service on Java 11.0.7 (Oracle Corporation) Windows Server 2016 10.0
2020-06-06 12:45:49.341 [elastic-apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: {  "build_date": "2020-05-12T00:04:54Z",  "build_sha": "64e91c95329991c36b16ff94fd34ea75230c06c2",  "version": "7.7.0"}
2020-06-06 12:45:49.341 [elastic-apm-remote-config-poller] INFO co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Received new configuration from APM Server: {enable_log_correlation=true}
2020-06-06 12:45:50.326 [Attach Listener] INFO co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2020-06-06 12:45:54.249 [restartedMain] INFO co.elastic.apm.agent.servlet.ServletVersionInstrumentation - Servlet container info = Apache Tomcat/9.0.33

Hi and thanks for your question!

Currently, you have to send your log files separately via Filebeat. Check out this guide: https://www.elastic.co/guide/en/apm/agent/java/current/log-correlation.html

But we're working on a solution where the agent would send the logs directly to APM Server so you don't have to do that manually. Subscribe to this issue to get notifications about the progress: https://github.com/elastic/apm-server/issues/3723

1 Like

Hi @felixbarny,

Thanks for the reply as suggested i have subscribed the issue open in github. I will try using the filebeat and let you know. Thanks

Hi @felixbarny,

I am having a question

In case if i use filebeat to ship my log4j logs how should APM knows this particular logs belongs to this particular request because i am going to trace my logs using Trace Logs option available in APM for each request.

What should be the key here to make APM to understand this log belongs to this request should i have to integrate filebeat to my java service? What is the approach here?

I am in little chaos could you please guide me on this because before start using filebeat i have to make sure these things.

There should be a trace.id in both the logs an the trace itself. The Trace logs action will get you to the Logs App and filter for that trace.id.

The easiest way to get started is ecs-logging-java: https://github.com/elastic/ecs-logging-java

Hi @felixbarny,

Thanks, I just followed the steps given in below link found it is working fine

logback-ecs-encoder

I mapped the json file which got created in the location to my filebeat which i specified in logback-spring.xml yes it is doing good.

When i look into the Trace Logs all the logs are displayed in Json format like below

{"@timestamp":"2020-06-13T16:18:07.668Z", "log.level": "INFO", "message":"UserId - J09786", "service.name":"FetchUser","event.dataset":"FetchUser.log","process.thread.name":"http-nio-8060-exec-6","log.logger":"com.UserController","transaction.id":"13e2041120ab51aa","trace.id":"9a425288690116672c493047e61125a7"}

Is there any possibility to use any other format like line by line log which exactly stored in log file like below?

2020-06-13 23:48:04 - UserId - J09786
2020-06-13 23:48:04 - Get the user details information from Fetch Service End
2020-06-13 23:48:49 - incoming text 1RjgMLaOLO5Z+Ya3qRlMWKGqj9hghfffxdff5634cDgQi/1QHWWojjlNdpe9S
2020-06-13 23:48:49 - result - http://localhost:8080/fetchuserdb
2020-06-13 23:48:49 - Decrypted auth token : xxxxxxxxxxxxxxxxx101

because i don't find any trace.id in log file, it is available only in Json file created. Kindly suggest.

You have to add the MDC to your logging pattern. See https://www.elastic.co/guide/en/apm/agent/java/current/log-correlation.html#log-correlation-extract-ids

1 Like

Ok, Sounds Good but as of now i have configured everything in central management and mapped the json file it's working fine.

Also raised another case regarding ECS logging. Kindly look into this. Thanks.

customizing-message-field-for-ecs-logging

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.