When I piloting elastic apm, I found that it is not showing the HTTP calls in the spans. It shows the MySQL queries but not the HTTP calls. I use microservices architecture and there are 50 services which calls other in someway. If this information is shown (Distributed Tracing), it will be really helpful and I can find out why the performance is not meeting. I use Spring Cloud libraries and OpenFeign for calling other microservices and Sleuth for adding trace & span info.
Hi and thanks for considering our APM offering!
It seems like Feign is using OkHttp by default. This HTTP client is currently not supported by the Java agent. See a list of all supported networking frameworks here.
We will however work on OkHttp support shortly. You can track the progress here: https://github.com/elastic/apm-agent-java/issues/329
Alternatively, you could configure feign to use the Apache HttpClient with this module: https://search.maven.org/artifact/com.netflix.feign/feign-httpclient/8.18.0/jar
Cheers,
Felix
Feign is Spring RestTemplate which in turn uses Apache HttpClient. I am using Spring Boot 2.0.5 & Spring Cloud Finchley.SR1. Here is the gradle dependencies,
| | +--- com.netflix.ribbon:ribbon-core:2.2.5 (*)
| | +--- com.netflix.ribbon:ribbon-httpclient:2.2.5
| | | +--- com.netflix.ribbon:ribbon-core:2.2.5 (*)
| | | +--- com.netflix.ribbon:ribbon-loadbalancer:2.2.5 (*)
| | | +--- commons-collections:commons-collections:3.2.2
| | | +--- org.apache.httpcomponents:httpclient:4.2.1 -> 4.5.6
| | | | +--- org.apache.httpcomponents:httpcore:4.4.10
| | | | \--- commons-codec:commons-codec:1.10 -> 1.11
| | | +--- com.sun.jersey:jersey-client:1.19.1
| | | | \--- com.sun.jersey:jersey-core:1.19.1
| | | | \--- javax.ws.rs:jsr311-api:1.1.1
| | | +--- com.sun.jersey.contribs:jersey-apache-client4:1.19.1
| | | | +--- org.apache.httpcomponents:httpclient:4.1.1 -> 4.5.6 (*)
| | | | \--- com.sun.jersey:jersey-client:1.19.1 (*)
| | | +--- org.slf4j:slf4j-api:1.7.12 -> 1.7.25
| | | +--- com.netflix.servo:servo-core:0.10.1 -> 0.12.21 (*)
| | | +--- com.google.guava:guava:16.0.1 -> 20.0
| | | +--- com.netflix.archaius:archaius-core:0.7.4 -> 0.7.6 (*)
| | | \--- com.netflix.netflix-commons:netflix-commons-util:0.1.1 -> 0.3.0 (*)
| | +--- com.netflix.ribbon:ribbon-loadbalancer:2.2.5 (*)
APM supported Apache HttpClient is 4.3+ and here used one is 4.5.6. I have unpacked the final built JAR and found that Apache HttpClient 4.5.6 is being used.
BTW, I am using Kotlin 1.3.0. Whether this might be the problem? I can see method names and DB instrumentation works fine.
Now I have added feign-httpclient-8.18, but still HTTP calls are not instrumented. Here is the dependencies tree,
runtime - Runtime dependencies for compilation 'main' (target (jvm)) (deprecated, use 'runtimeOnly ' instead).
\--- com.netflix.feign:feign-httpclient:8.18.0
+--- com.netflix.feign:feign-core:8.18.0
| \--- org.jvnet:animal-sniffer-annotation:1.0
\--- org.apache.httpcomponents:httpclient:4.5.1 -> 4.5.6
+--- org.apache.httpcomponents:httpcore:4.4.10
\--- commons-codec:commons-codec:1.10 -> 1.11
| +--- org.springframework.cloud:spring-cloud-starter-netflix-ribbon:2.0.1.RELEASE
| | +--- org.springframework.cloud:spring-cloud-starter:2.0.1.RELEASE (*)
| | +--- org.springframework.cloud:spring-cloud-netflix-ribbon:2.0.1.RELEASE (*)
| | +--- org.springframework.cloud:spring-cloud-starter-netflix-archaius:2.0.1.RELEASE (*)
| | +--- com.netflix.ribbon:ribbon:2.2.5
| | | +--- com.netflix.ribbon:ribbon-core:2.2.5
| | | | +--- org.slf4j:slf4j-api:1.6.4 -> 1.7.25
| | | | +--- com.google.guava:guava:16.0 -> 20.0
| | | | +--- commons-configuration:commons-configuration:1.8 (*)
| | | | +--- commons-lang:commons-lang:2.6
| | | | \--- com.netflix.archaius:archaius-core:0.7.4 -> 0.7.6 (*)
| | | +--- com.netflix.ribbon:ribbon-transport:2.2.5
| | | | +--- com.netflix.ribbon:ribbon-core:2.2.5 (*)
| | | | +--- com.netflix.ribbon:ribbon-loadbalancer:2.2.5
| | | | | +--- com.netflix.ribbon:ribbon-core:2.2.5 (*)
| | | | | +--- com.netflix.netflix-commons:netflix-statistics:0.1.1
| | | | | | \--- org.slf4j:slf4j-api:1.6.4 -> 1.7.25
| | | | | +--- io.reactivex:rxjava:1.0.9 -> 1.2.0
| | | | | +--- org.slf4j:slf4j-api:1.7.12 -> 1.7.25
| | | | | +--- com.netflix.servo:servo-core:0.10.1 -> 0.12.21 (*)
| | | | | +--- com.google.guava:guava:16.0.1 -> 20.0
| | | | | +--- com.netflix.archaius:archaius-core:0.7.4 -> 0.7.6 (*)
| | | | | \--- com.netflix.netflix-commons:netflix-commons-util:0.1.1 -> 0.3.0
| | | | | +--- org.slf4j:slf4j-api:1.6.4 -> 1.7.25
| | | | | \--- javax.inject:javax.inject:1
| | | | +--- io.reactivex:rxjava:1.0.10 -> 1.2.0
| | | | +--- io.reactivex:rxnetty:0.4.9
| | | | | +--- io.reactivex:rxjava:1.0.10 -> 1.2.0
| | | | | +--- io.netty:netty-codec-http:4.0.27.Final -> 4.1.29.Final
| | | | | | \--- io.netty:netty-codec:4.1.29.Final
| | | | | | \--- io.netty:netty-transport:4.1.29.Final
| | | | | | +--- io.netty:netty-buffer:4.1.29.Final
| | | | | | | \--- io.netty:netty-common:4.1.29.Final
| | | | | | \--- io.netty:netty-resolver:4.1.29.Final
| | | | | | \--- io.netty:netty-common:4.1.29.Final
| | | | | +--- io.netty:netty-transport-native-epoll:4.0.27.Final -> 4.1.29.Final
| | | | | | +--- io.netty:netty-common:4.1.29.Final
| | | | | | +--- io.netty:netty-buffer:4.1.29.Final (*)
| | | | | | +--- io.netty:netty-transport-native-unix-common:4.1.29.Final
| | | | | | | +--- io.netty:netty-common:4.1.29.Final
| | | | | | | \--- io.netty:netty-transport:4.1.29.Final (*)
| | | | | | \--- io.netty:netty-transport:4.1.29.Final (*)
| | | | | \--- org.slf4j:slf4j-api:1.7.6 -> 1.7.25
| | | | +--- io.reactivex:rxnetty-contexts:0.4.9
| | | | | +--- io.reactivex:rxjava:1.0.10 -> 1.2.0
| | | | | \--- io.reactivex:rxnetty:0.4.9 (*)
| | | | +--- io.reactivex:rxnetty-servo:0.4.9
| | | | | +--- io.reactivex:rxjava:1.0.10 -> 1.2.0
| | | | | +--- io.reactivex:rxnetty:0.4.9 (*)
| | | | | \--- com.netflix.servo:servo-core:0.7.5 -> 0.12.21 (*)
| | | | +--- javax.inject:javax.inject:1
| | | | +--- org.slf4j:slf4j-api:1.7.12 -> 1.7.25
| | | | +--- com.google.guava:guava:16.0.1 -> 20.0
| | | | \--- com.netflix.archaius:archaius-core:0.7.4 -> 0.7.6 (*)
| | | +--- com.netflix.hystrix:hystrix-core:1.4.3 -> 1.5.12
| | | | +--- org.slf4j:slf4j-api:1.7.0 -> 1.7.25
| | | | +--- com.netflix.archaius:archaius-core:0.4.1 -> 0.7.6 (*)
| | | | +--- io.reactivex:rxjava:1.2.0
| | | | \--- org.hdrhistogram:HdrHistogram:2.1.9 -> 2.1.10
| | | +--- javax.inject:javax.inject:1
| | | +--- io.reactivex:rxjava:1.0.10 -> 1.2.0
| | | +--- io.reactivex:rxnetty:0.4.9 (*)
| | | +--- commons-configuration:commons-configuration:1.8 (*)
| | | +--- com.google.guava:guava:16.0.1 -> 20.0
| | | \--- com.netflix.archaius:archaius-core:0.7.4 -> 0.7.6 (*)
| | +--- com.netflix.ribbon:ribbon-core:2.2.5 (*)
| | +--- com.netflix.ribbon:ribbon-httpclient:2.2.5
| | | +--- com.netflix.ribbon:ribbon-core:2.2.5 (*)
| | | +--- com.netflix.ribbon:ribbon-loadbalancer:2.2.5 (*)
| | | +--- commons-collections:commons-collections:3.2.2
| | | +--- org.apache.httpcomponents:httpclient:4.2.1 -> 4.5.6 (*)
| | | +--- com.sun.jersey:jersey-client:1.19.1 (*)
| | | +--- com.sun.jersey.contribs:jersey-apache-client4:1.19.1 (*)
| | | +--- org.slf4j:slf4j-api:1.7.12 -> 1.7.25
| | | +--- com.netflix.servo:servo-core:0.10.1 -> 0.12.21 (*)
| | | +--- com.google.guava:guava:16.0.1 -> 20.0
| | | +--- com.netflix.archaius:archaius-core:0.7.4 -> 0.7.6 (*)
| | | \--- com.netflix.netflix-commons:netflix-commons-util:0.1.1 -> 0.3.0 (*)Could you try to find out which HTTP client is effectively used? The auto-configuration chooses different clients based on what's available on the class path see FeignRibbonClientAutoConfiguration and FeignAutoConfiguration. The easiest would be to debug a remote call and to see which HTTP client is actually handling it.
If it is indeed executed by a supported HTTP client and it's still not working, could you enable the debug log level of the agent and send me the full agent log from startup of the application to one remote execution of feign?
Thanks,
Felix
I also tried with http-client (using it directly) without getting anything. My manual instrumented spans are displayed in kibana though. I can see in the apm-server log that nothing gets to the apm-server with auto instrumenting, only with manual instrumentation.
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.6</version>
</dependency>
CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpGet = new HttpGet("http://google.se");
CloseableHttpResponse response = httpclient.execute(httpGet);
System.out.println(response.getStatusLine());
Thanks /Johan
I have debugged the client and found that it is using ApacheHttpClient. Here is the screenshot,
Here are the debug logs
Kindly let me know if I am doing something wrong.
Note that spans will only be created when there is an enclosing transaction for that. Could that be a problem in your case? I'll have a look at the logs in a bit.
Nope, that gives me nothing.
I have to code the instrumentation to get something into APM Server
Transaction transaction = ElasticApm.startTransaction();
Oh, I know what the problem is. The Apache HttpClient instrumentation is still marked as incubating by mistake! You can enable it by setting disable_instrumentations to none.
Sorry for that. Please tell me if that helped.
1 Like
Thanks @felixbarny. After adding compile("com.netflix.feign:feign-httpclient:8.18.0") and setting disable_instrumentations to none, spans are getting added for HTTP calls. 
1 Like
There is an issue with com.netflix.feign:feign-httpclient:8.18.0, so use io.github.openfeign:feign-httpclient:9.5.1
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.