What Kind of logs will push to ES via APM

[Sajid_Md]

Dear All,

I have some queries related to APM server to ES via APM agent.

Kibana version: 6.2

Elasticsearch version: 6.2

APM Server version: 6.5

APM Agent language and version: ES APM agent 1.5

But In Kibana dashboard APM tab was not added, it's showing in dashboard list like below. How to add APM as a separate tab in Kibana dashboard.

Also, I have a few questions:

• Which type of data will be pushing into APM server from agent applications.
• For example, I added an agent for one application that will take the input from one application and insert into DB servers. here which type of logs I can expect into APM server?
• In the APM dashboard, I can see a total of 5 tabs for APM in that Span details is not showing for my applications.
• Error details and Error dashboard tabs will show when we are having errors from applications?
• How to add the main APM tab in my Kibana dashboard like below.
  

Could you please help me resolve my concerns.

Thanks in advance...

[axw (Andrew Wilkins)]

@Sajid_Md

But In Kibana dashboard APM tab was not added, it's showing in dashboard list like below. How to add APM as a separate tab in Kibana dashboard.

For Kibana 6.2, the APM UI is part of X-Pack, which you must install as a separate step: Installing X-Pack in Kibana | Kibana User Guide [6.2] | Elastic. In more recent versions (6.3+) of the stack, the APM UI is part of the default distribution.

@Sajid_Md

Also, I have a few questions:

• Which type of data will be pushing into APM server from agent applications.
• For example, I added an agent for one application that will take the input from one application and insert into DB servers. here which type of logs I can expect into APM server?

You can find a list of the Java agent's supported technologies here: Supported technologies | APM Java Agent Reference [1.x] | Elastic.

If your application receives input over HTTP, and uses one of the supported frameworks, JVMs, and application servers: then each incoming HTTP request will be reported as a transaction to Elastic APM. If you're using JDBC 4.1+ with a driver compiled for Java 1.5+, then each database operation will be reported as a span.

@Sajid_Md

• In the APM dashboard, I can see a total of 5 tabs for APM in that Span details is not showing for my applications.

I suggest you read through Troubleshooting | APM Java Agent Reference [1.x] | Elastic.

@Sajid_Md

• Error details and Error dashboard tabs will show when we are having errors from applications?

Yes, for example if an exception is raised from within a ServletHandler, then the agent will capture that and report it to Elastic APM.

[Sajid_Md]

Thanks, @axw for your valuable response.

@axw

Sajid_Md:

I have added agent jar for one application which will receive the traffic from one application and insert into MYSQL server. but here I can able to span details. also in apm server log, I can see only info logs pushing to the server. Below are the logs (correct me if am wrong as per apm server log)

========
2019-04-15T10:19:32.603+0500 INFO [request] beater/common_handlers.go:272 handled request {"request_id": "1d21543d-3fa1-41af-8d98-c7d34a69452c", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "10.176.11.39", "user-agent": "java-agent/1.5.0", "response_code": 202}
2019-04-15T10:19:42.886+0500 INFO [request] beater/common_handlers.go:272 handled request {"request_id": "10656613-7c21-4175-a5f3-26b98a2bd991", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "10.176.11.39", "user-agent": "java-agent/1.5.0", "response_code": 202}

===

Also do we need to modify any applicaiton code to push data from application to APM serrver. ?

[axw (Andrew Wilkins)]

@Sajid_Md

I have added agent jar for one application which will receive the traffic from one application and insert into MYSQL server. but here I can able to span details. also in apm server log, I can see only info logs pushing to the server. Below are the logs (correct me if am wrong as per apm server log)

If you can't see the spans, then my suggestion remains: read through Troubleshooting | APM Java Agent Reference [1.x] | Elastic. You may be able to save some time by checking that you're using the supported technologies first.

@Sajid_Md

Also do we need to modify any applicaiton code to push data from application to APM serrver. ?

Assuming you're using supported technologies, no.

[Sajid_Md]

Hi @axw,

Thanks for your support @axw.

@axw

If you can't see the spans, then my suggestion remains: read through https://www.elastic.co/guide/en/apm/agent/java/current/trouble-shooting.html . You may be able to save some time by checking that you're using the supported technologies first.

I have gone through the provided link. while starting my application i can see only INFO logs are loading to APM server (as per logs). but where i can set all the logs to upload into APM server. here am uploading only elastic-apm-agent-1.5.0.jar using javaagent. below is my executable file configuration.

export JAVA_OPTS="$JAVA_OPTS -Delastic.apm.service_name=MLogServerJ -Delastic.apm.server_urls=http://10.176.11.39:9210 -Delastic.apm.application_packages=org.example"
exec /usr/java/jdk1.8.0_91/bin/java $JAVA_OPTS -javaagent:./lib/elastic-apm-agent-1.5.0.jar -Dcomponent=mLogserverJ -Xmx3g -classpath ${CLASSPATH} com.imi.mlogserverj.MLogServerJ "/apps/imidigital/mlogserverj/mlogserverj.properties"
2019-04-16 12:34:51.346 [apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: {"ok":{"build_date":"2018-11-09T17:48:30Z","build_sha":"2c46a2919dbe06767caeacbffd0055aab1493991","version":"6.5.0"}}
2019-04-16 12:34:51.381 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.5.0 as MLogServerJ on Java 1.8.0_91 (Oracle Corporation) Linux 3.10.0-514.el7.x86_64

please help me here to push all logs related to that application into APM server.

[felixbarny (Felix Barnsteiner)]

@Sajid_Md

please help me here to push all logs related to that application into APM server.

Hi Sajid,

our agent don't support sending their logs to the APM Server. They only send traces and metrics to the APM Server.

Is your question about how to ingest your application logs to Elasticsearch? If so, take a look at Filebeat.

Or are you asking why you don't see traces in the APM UI? If so, please tell me which application server and frameworks you are using. Have you checked that they are listed under the supported technologies? If that matches, please set the log_level to DEBUG, execute a request against your application and paste the debug logs here.

[Sajid_Md]

HI @felixbarny,

Thank you for comments.

below is the snapshot of my application logs, which is having info, debug and error logs enable in my application level.

Java APM agent - -javaagent --> elastic-apm-agent-1.5.0.jar
Using Java Version - "1.8.0_91"'

Also here do we have option to enable all types of logs in logging.level: prop in apm-server.yml config file.
i tried to enable all with comma separate and multiple ways but not worked. could you please help here.

[felixbarny (Felix Barnsteiner)]

I can't spot any agent logs here. It's also usually better to upload the full text of the logs, for example via gist.github.com.

Asking again: which application server and web frameworks are you using?