Not able to see the complete trace except the root request

sattishv · January 14, 2019, 2:58pm

Hi team,

We were able to configure the agent and the server by modifying the localhost to 0.0.0.0 in config files since we use remote apm-server. After configuration we could see only the root method not the complete trace we couldn't drill down further from the root transaction. Do we need to add add the packages list. Have attached the screenshot

JAVA options used:

rapper.java.additional.46=-javaagent:/opt/elastic-apm-agent-1.2.0.jar
wrapper.java.additional.47=-Delastic.apm.service_name=DemoAPP
wrapper.java.additional.48=-Delastic.apm.server_url=http://10.239.39.83:8200

Thanks
Sattish

felixbarny · January 14, 2019, 6:57pm

Hi and thanks for trying out the Java agent!

Our agents don't trace all the methods of your application by default. By default we mostly capture external calls. See a full list of supported technologies here: https://www.elastic.co/guide/en/apm/agent/java/current/supported-technologies-details.html

But with 1.3.0, you can configure which methods of your application should be traced: https://www.elastic.co/guide/en/apm/agent/java/current/config-core.html#config-trace-methods

Cheers,
Felix

sattishv · January 16, 2019, 5:31pm

Hi Felix,

Thanks for the response. Have few queries based on your inputs. Our platform uses Microservices predominantly using REST style with JAX-RS. I see those been supported in your list.But still i couldn't see the complete trace. Not sure what am i missing in my configuration.

Does elastic apm support openjdk 1.8 versions

Thanks
Sattish.

felixbarny · January 16, 2019, 6:59pm

What exactly is it you are missing? Is the transaction recorded? Does it have the correct name? Which HTTP client are you using to call the other services? Are those calls collected?

Note that the Java agent currently supports these HTTP clients: Supported technologies | APM Java Agent Reference [1.x] | Elastic. So you might need to reconfigure your JAX-RS client to use one of those.

Yes, see Supported technologies | APM Java Agent Reference [1.x] | Elastic

Cheers,
Felix

sattishv · January 18, 2019, 11:55am

Hi Felix,

Verified the agent supported technologies. We are using the right Apache Http client 4.5.6. JAX_RS specification and Resteasy as implementation of JAX-RS classes. Still i am not able to trace. Can you let us where the agent logs are stored.

Sattish.

felixbarny · January 18, 2019, 12:32pm

See this section about troubleshooting and logging: https://www.elastic.co/guide/en/apm/agent/java/current/trouble-shooting.html#trouble-shooting-logging

Could you attach your debug logs of starting up the application and executing an jax-rs http request once?

Thanks,
Felix

sattishv · January 19, 2019, 6:52am

Hi Felix,

Thanks. It looks like i haven't set the agent configuration file . My agent home has only agent.jar . I believe thats the reason i couldn't the agent logs separately. Do we need to explicitly create the agent configuration file at location where the agent jar resides. Could you please confirm.

Sattish.

felixbarny · January 19, 2019, 8:53am

Here are multiple ways to configure the agent. See the configuration docs for details.

Do we need to explicitly create the agent configuration file at location where the agent jar resides.

Yes, this is one option of configuring the agent.

sattishv · January 19, 2019, 9:18am

Hi Felix,

I could generate the agent logs by configuring the log_level to TRACE but not able to attach since upload option allows only image files but mine has a extn .log. Let me know the other ways

Sattish.

felixbarny · January 19, 2019, 9:30am

You can use https://gist.github.com/ for instance.

sattishv · January 19, 2019, 2:59pm

Hi felix,

I have uploaded the agent log to my gist

gist.github.com

https://gist.github.com/sattishv/94c118749033711a14b37b7d208395b0

elastic-apm.log

2019-01-19 00:47:25.139 [apm-server-healthcheck] DEBUG co.elastic.apm.agent.report.ApmServerHealthChecker - Starting healthcheck to http://10.180.176.96:8200/
2019-01-19 00:47:25.202 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.2.0 as fimanagementserv-ca on Java 1.8.0-zing_18.08.0.0 (Azul Systems, Inc.) Linux 4.4.0-133-generic
2019-01-19 00:47:25.202 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - service_name: 'fimanagementserv-ca' (source: Java System Properties)
2019-01-19 00:47:25.202 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - server_urls: 'http://10.180.176.96:8200' (source: Java System Properties)
2019-01-19 00:47:25.202 [main] WARN co.elastic.apm.agent.configuration.StartupInfo - Detected usage of an old configuration key: 'server_url'. Please use 'server_urls' instead.
2019-01-19 00:47:25.202 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - log_level: 'TRACE' (source: Java System Properties)
2019-01-19 00:47:25.203 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - log_file: '/opt/logs/elastic-apm.log' (source: Java System Properties)
2019-01-19 00:47:25.203 [main] WARN co.elastic.apm.agent.configuration.StartupInfo - To enable all features and to increase startup times, please configure application_packages
2019-01-19 00:47:25.220 [apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: {"ok":{"build_date":"2018-12-17T20:15:18Z","build_sha":"fd4f1ac006d8896e4957b6c01fb1e9b9ff04afd7","version":"6.5.4"}}
2019-01-19 00:47:25.236 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying advice co.elastic.apm.agent.httpclient.ApacheHttpClientInstrumentation

This file has been truncated. show original

Thanks
Sattish

felixbarny · January 19, 2019, 5:39pm

Thanks for providing the logs.

I could see that the Apache HTTP client does get successfully instrumented. But I don't see a span created for outgoing HTTP requests. The reason for might be that your JAX-RS client does not use the Apache HTTP client. Note that the agent does not support Apache's async client yet. Could you confirm which HTTP client is actually used? Preferably by stepping through the code with a debugger.

Thanks,
Felix

sattishv · January 22, 2019, 9:40am

Hi Felix,

Had a conversation with my Dev team. They confirmed that JAX-RS uses apache http client 4.5.6 for sync and for async they use RxNetty

Thanks
Sattish

felixbarny · January 22, 2019, 11:00am

Hi Sattish,

So maybe there is an async call in this case?
That would explain why there is no span for it. RxNetty is currently not supported.

Thanks,
Felix

felixbarny · January 22, 2019, 11:03am

You could use the createSpan and injectTraceHeaders APIs to manually create a span for the HTTP call to transfer the trace context in the mean time.

system · February 12, 2019, 7:03am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.