Traces are not shown in Kibana

Elastic Observability APM
, ,
22

This looks like you're having issues with your docker networking. I recommend docker exec'ing into the container to make sure you're able to reach the apm-server container with cURL or something similar.

23

Hi @stuart.nelson, APM server is accessible from my application container when I execute curl -I host.docker.internal:8200 from my docker container, following response is returned:

HTTP/1.1 200 OK
Content-Type: application/json
X-Content-Type-Options: nosniff
Date: Wed, 09 Feb 2022 10:26:35 GMT
Content-Length: 150

{
  "build_date": "2022-01-06T23:13:12Z",
  "build_sha": "fde0af4fa2b9f39e518b333c5be56cf8be215ca0",
  "publish_ready": true,
  "version": "7.16.3"
}
24

Hi @stuart.nelson Any update?

25

@rishisalunkhe let's back up a bit. I can see in your agent log file that the Java Agent is sending requests to APM Server every 30 seconds:

$ grep Starting agentlogs.txt 
2022-02-03 09:22:18,763 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Starting new request to http://host.docker.internal:8200/intake/v2/events
2022-02-03 09:22:48,728 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Starting new request to http://host.docker.internal:8200/intake/v2/events
2022-02-03 09:23:18,695 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Starting new request to http://host.docker.internal:8200/intake/v2/events
2022-02-03 09:23:48,657 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Starting new request to http://host.docker.internal:8200/intake/v2/events
2022-02-03 09:24:18,627 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Starting new request to http://host.docker.internal:8200/intake/v2/events
2022-02-03 09:24:48,593 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Starting new request to http://host.docker.internal:8200/intake/v2/events

This lines up with the default metrics interval: Reporter configuration options | APM Java Agent Reference [1.x] | Elastic

I suspect the issue is that the Java agent isn't sending any trace events. Please have a look at Supported technologies | APM Java Agent Reference [1.x] | Elastic, and confirm that you are using supported technologies.

26

@axw Thanks for your reply, I checked and I am using supported technologies as below:
OpenJDK Runtime Environment Corretto-11.0.3.7.1
Tomcat 8.5.33.0
Servlet 3.0.1

Application running inside docker, ES and Kibana is also in docker, APM server is running on Mac, APM-agent is placed inside /bin directory of tomcat

27

The agent debug log is just saying that it is checking the server for new config. There doesn't seem to be any traceable activity happening in the Java application for it to send traces

28

@Jack_Shirazi thanks, but I saw the methods are found for traces in agent log file

29

but I saw the methods are found for traces in agent log file

Not sure what this means - do you mean there are method matches? That's not enough, there also needs to be an invocation of the instrumented method.
Please share a full description of how you installed the agent on your Tomcat and the full agent configurations you use. In addition, a useful debug log would include the startup of the Servlet container and until after some Servlet invocations took place.

30

Hi @Eyal_Koren , Thanks for your reply, my application is running inside the docker on tomcat server and I placed APM-agent jar in /bin directory of tomcat. APM server is running on MAC. Below is the configuration: Also attached agent logs in gist:

export CATALINA_OPTS='-server -Xms64M -Xmx512M -XX:+UseConcMarkSweepGC -XX:-OmitStackTraceInFastThrow -Dcom.sun.management.jmxremote.port=9014 -Dcom.sun.management.jmxremote.password.file=/usr/local/test-app/jmxremote.password -Dcom.sun.management.jmxremote.access.file=/usr/local/test-app/jmxremote.access -Dcom.sun.management.jmxremote.ssl=false -Dcatalina.base=/usr/local/test-app -agentlib:jdwp=transport=dt_socket,address=*:5005,server=y,suspend=n -javaagent:/usr/local/test-app/bin/elastic-apm-agent-1.28.4.jar -Delastic.apm.service_name=my-application -Delastic.apm.server_url=http://host.docker.internal:8200 -Delastic.apm.log_level=DEBUG'
31

@Eyal_Koren any update on my query?

32

The truncated debug log you shared is insufficient for further analysis

33

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.