# Distributed tracing - HTTP calls are not shown in span

**URL:** <https://discuss.elastic.co/t/distributed-tracing-http-calls-are-not-shown-in-span/158302>\
**Category:** APM\
**Created:** [November 27, 2018, 8:58am UTC](https://discuss.elastic.co/t/distributed-tracing-http-calls-are-not-shown-in-span/158302 "2018-11-27T08:58:12Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![jaks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaks/32/38097_2.png) [@jaks](https://discuss.elastic.co/u/jaks)\
**Post date:** [November 27, 2018, 8:58am UTC](https://discuss.elastic.co/t/distributed-tracing-http-calls-are-not-shown-in-span/158302/1 "2018-11-27T08:58:12Z")

</div>

When I piloting elastic apm, I found that it is not showing the HTTP calls in the spans. It shows the MySQL queries but not the HTTP calls. I use microservices architecture and there are 50 services which calls other in someway. If this information is shown (Distributed Tracing), it will be really helpful and I can find out why the performance is not meeting. I use [Spring Cloud](http://spring.io/projects/spring-cloud) libraries and [OpenFeign](https://search.maven.org/search?q=a:spring-cloud-starter-openfeign) for calling other microservices and [Sleuth](https://search.maven.org/search?q=a:spring-cloud-starter-sleuth) for adding trace & span info.

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [November 27, 2018, 9:29am UTC](https://discuss.elastic.co/t/distributed-tracing-http-calls-are-not-shown-in-span/158302/2 "2018-11-27T09:29:17Z")

</div>

Hi and thanks for considering our APM offering!

It seems like Feign is using OkHttp by default. This HTTP client is currently not supported by the Java agent. See a list of all supported networking frameworks [here](https://www.elastic.co/guide/en/apm/agent/java/current/supported-technologies-details.html#supported-networking-frameworks).

We will however work on OkHttp support shortly. You can track the progress here: [https://github.com/elastic/apm-agent-java/issues/329](https://github.com/elastic/apm-agent-java/issues/329)

Alternatively, you could configure feign to use the Apache HttpClient with this module: [https://search.maven.org/artifact/com.netflix.feign/feign-httpclient/8.18.0/jar](https://search.maven.org/artifact/com.netflix.feign/feign-httpclient/8.18.0/jar)

Cheers,  
Felix

---

<div class="post-metadata">

**Author:** ![jaks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaks/32/38097_2.png) [@jaks](https://discuss.elastic.co/u/jaks)\
**Post date:** [November 27, 2018, 10:03am UTC](https://discuss.elastic.co/t/distributed-tracing-http-calls-are-not-shown-in-span/158302/3 "2018-11-27T10:03:10Z")

</div>

Feign is Spring RestTemplate which in turn uses Apache HttpClient. I am using Spring Boot 2.0.5 & Spring Cloud Finchley.SR1. Here is the gradle dependencies,

```
| | +--- com.netflix.ribbon:ribbon-core:2.2.5 (*)
| | +--- com.netflix.ribbon:ribbon-httpclient:2.2.5
| | | +--- com.netflix.ribbon:ribbon-core:2.2.5 (*)
| | | +--- com.netflix.ribbon:ribbon-loadbalancer:2.2.5 (*)
| | | +--- commons-collections:commons-collections:3.2.2
| | | +--- org.apache.httpcomponents:httpclient:4.2.1 -> 4.5.6
| | | | +--- org.apache.httpcomponents:httpcore:4.4.10
| | | | \--- commons-codec:commons-codec:1.10 -> 1.11
| | | +--- com.sun.jersey:jersey-client:1.19.1
| | | | \--- com.sun.jersey:jersey-core:1.19.1
| | | | \--- javax.ws.rs:jsr311-api:1.1.1
| | | +--- com.sun.jersey.contribs:jersey-apache-client4:1.19.1
| | | | +--- org.apache.httpcomponents:httpclient:4.1.1 -> 4.5.6 (*)
| | | | \--- com.sun.jersey:jersey-client:1.19.1 (*)
| | | +--- org.slf4j:slf4j-api:1.7.12 -> 1.7.25
| | | +--- com.netflix.servo:servo-core:0.10.1 -> 0.12.21 (*)
| | | +--- com.google.guava:guava:16.0.1 -> 20.0
| | | +--- com.netflix.archaius:archaius-core:0.7.4 -> 0.7.6 (*)
| | | \--- com.netflix.netflix-commons:netflix-commons-util:0.1.1 -> 0.3.0 (*)
| | +--- com.netflix.ribbon:ribbon-loadbalancer:2.2.5 (*)

```

APM supported Apache HttpClient is 4.3+ and here used one is 4.5.6. I have unpacked the final built JAR and found that [Apache HttpClient 4.5.6](https://search.maven.org/artifact/org.apache.httpcomponents/httpclient/4.5.6/jar) is being used.

BTW, I am using Kotlin 1.3.0. Whether this might be the problem? I can see method names and DB instrumentation works fine.

---

<div class="post-metadata">

**Author:** ![jaks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaks/32/38097_2.png) [@jaks](https://discuss.elastic.co/u/jaks)\
**Post date:** [November 27, 2018, 10:50am UTC](https://discuss.elastic.co/t/distributed-tracing-http-calls-are-not-shown-in-span/158302/4 "2018-11-27T10:50:55Z")

</div>

Now I have added [feign-httpclient-8.18](https://search.maven.org/artifact/com.netflix.feign/feign-httpclient/8.18.0/jar), but still HTTP calls are not instrumented. Here is the dependencies tree,

```
runtime - Runtime dependencies for compilation 'main' (target (jvm)) (deprecated, use 'runtimeOnly ' instead).
\--- com.netflix.feign:feign-httpclient:8.18.0
     +--- com.netflix.feign:feign-core:8.18.0
     | \--- org.jvnet:animal-sniffer-annotation:1.0
     \--- org.apache.httpcomponents:httpclient:4.5.1 -> 4.5.6
          +--- org.apache.httpcomponents:httpcore:4.4.10
          \--- commons-codec:commons-codec:1.10 -> 1.11

| +--- org.springframework.cloud:spring-cloud-starter-netflix-ribbon:2.0.1.RELEASE
| | +--- org.springframework.cloud:spring-cloud-starter:2.0.1.RELEASE (*)
| | +--- org.springframework.cloud:spring-cloud-netflix-ribbon:2.0.1.RELEASE (*)
| | +--- org.springframework.cloud:spring-cloud-starter-netflix-archaius:2.0.1.RELEASE (*)
| | +--- com.netflix.ribbon:ribbon:2.2.5
| | | +--- com.netflix.ribbon:ribbon-core:2.2.5
| | | | +--- org.slf4j:slf4j-api:1.6.4 -> 1.7.25
| | | | +--- com.google.guava:guava:16.0 -> 20.0
| | | | +--- commons-configuration:commons-configuration:1.8 (*)
| | | | +--- commons-lang:commons-lang:2.6
| | | | \--- com.netflix.archaius:archaius-core:0.7.4 -> 0.7.6 (*)
| | | +--- com.netflix.ribbon:ribbon-transport:2.2.5
| | | | +--- com.netflix.ribbon:ribbon-core:2.2.5 (*)
| | | | +--- com.netflix.ribbon:ribbon-loadbalancer:2.2.5
| | | | | +--- com.netflix.ribbon:ribbon-core:2.2.5 (*)
| | | | | +--- com.netflix.netflix-commons:netflix-statistics:0.1.1
| | | | | | \--- org.slf4j:slf4j-api:1.6.4 -> 1.7.25
| | | | | +--- io.reactivex:rxjava:1.0.9 -> 1.2.0
| | | | | +--- org.slf4j:slf4j-api:1.7.12 -> 1.7.25
| | | | | +--- com.netflix.servo:servo-core:0.10.1 -> 0.12.21 (*)
| | | | | +--- com.google.guava:guava:16.0.1 -> 20.0
| | | | | +--- com.netflix.archaius:archaius-core:0.7.4 -> 0.7.6 (*)
| | | | | \--- com.netflix.netflix-commons:netflix-commons-util:0.1.1 -> 0.3.0
| | | | | +--- org.slf4j:slf4j-api:1.6.4 -> 1.7.25
| | | | | \--- javax.inject:javax.inject:1
| | | | +--- io.reactivex:rxjava:1.0.10 -> 1.2.0
| | | | +--- io.reactivex:rxnetty:0.4.9
| | | | | +--- io.reactivex:rxjava:1.0.10 -> 1.2.0
| | | | | +--- io.netty:netty-codec-http:4.0.27.Final -> 4.1.29.Final
| | | | | | \--- io.netty:netty-codec:4.1.29.Final
| | | | | | \--- io.netty:netty-transport:4.1.29.Final
| | | | | | +--- io.netty:netty-buffer:4.1.29.Final
| | | | | | | \--- io.netty:netty-common:4.1.29.Final
| | | | | | \--- io.netty:netty-resolver:4.1.29.Final
| | | | | | \--- io.netty:netty-common:4.1.29.Final
| | | | | +--- io.netty:netty-transport-native-epoll:4.0.27.Final -> 4.1.29.Final
| | | | | | +--- io.netty:netty-common:4.1.29.Final
| | | | | | +--- io.netty:netty-buffer:4.1.29.Final (*)
| | | | | | +--- io.netty:netty-transport-native-unix-common:4.1.29.Final
| | | | | | | +--- io.netty:netty-common:4.1.29.Final
| | | | | | | \--- io.netty:netty-transport:4.1.29.Final (*)
| | | | | | \--- io.netty:netty-transport:4.1.29.Final (*)
| | | | | \--- org.slf4j:slf4j-api:1.7.6 -> 1.7.25
| | | | +--- io.reactivex:rxnetty-contexts:0.4.9
| | | | | +--- io.reactivex:rxjava:1.0.10 -> 1.2.0
| | | | | \--- io.reactivex:rxnetty:0.4.9 (*)
| | | | +--- io.reactivex:rxnetty-servo:0.4.9
| | | | | +--- io.reactivex:rxjava:1.0.10 -> 1.2.0
| | | | | +--- io.reactivex:rxnetty:0.4.9 (*)
| | | | | \--- com.netflix.servo:servo-core:0.7.5 -> 0.12.21 (*)
| | | | +--- javax.inject:javax.inject:1
| | | | +--- org.slf4j:slf4j-api:1.7.12 -> 1.7.25
| | | | +--- com.google.guava:guava:16.0.1 -> 20.0
| | | | \--- com.netflix.archaius:archaius-core:0.7.4 -> 0.7.6 (*)
| | | +--- com.netflix.hystrix:hystrix-core:1.4.3 -> 1.5.12
| | | | +--- org.slf4j:slf4j-api:1.7.0 -> 1.7.25
| | | | +--- com.netflix.archaius:archaius-core:0.4.1 -> 0.7.6 (*)
| | | | +--- io.reactivex:rxjava:1.2.0
| | | | \--- org.hdrhistogram:HdrHistogram:2.1.9 -> 2.1.10
| | | +--- javax.inject:javax.inject:1
| | | +--- io.reactivex:rxjava:1.0.10 -> 1.2.0
| | | +--- io.reactivex:rxnetty:0.4.9 (*)
| | | +--- commons-configuration:commons-configuration:1.8 (*)
| | | +--- com.google.guava:guava:16.0.1 -> 20.0
| | | \--- com.netflix.archaius:archaius-core:0.7.4 -> 0.7.6 (*)
| | +--- com.netflix.ribbon:ribbon-core:2.2.5 (*)
| | +--- com.netflix.ribbon:ribbon-httpclient:2.2.5
| | | +--- com.netflix.ribbon:ribbon-core:2.2.5 (*)
| | | +--- com.netflix.ribbon:ribbon-loadbalancer:2.2.5 (*)
| | | +--- commons-collections:commons-collections:3.2.2
| | | +--- org.apache.httpcomponents:httpclient:4.2.1 -> 4.5.6 (*)
| | | +--- com.sun.jersey:jersey-client:1.19.1 (*)
| | | +--- com.sun.jersey.contribs:jersey-apache-client4:1.19.1 (*)
| | | +--- org.slf4j:slf4j-api:1.7.12 -> 1.7.25
| | | +--- com.netflix.servo:servo-core:0.10.1 -> 0.12.21 (*)
| | | +--- com.google.guava:guava:16.0.1 -> 20.0
| | | +--- com.netflix.archaius:archaius-core:0.7.4 -> 0.7.6 (*)
| | | \--- com.netflix.netflix-commons:netflix-commons-util:0.1.1 -> 0.3.0 (*)
```

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [November 27, 2018, 10:57am UTC](https://discuss.elastic.co/t/distributed-tracing-http-calls-are-not-shown-in-span/158302/5 "2018-11-27T10:57:36Z")

</div>

Could you try to find out which HTTP client is effectively used? The auto-configuration chooses different clients based on what's available on the class path see [FeignRibbonClientAutoConfiguration](https://github.com/spring-cloud/spring-cloud-openfeign/blob/master/spring-cloud-openfeign-core/src/main/java/org/springframework/cloud/openfeign/ribbon/FeignRibbonClientAutoConfiguration.java) and [FeignAutoConfiguration](https://github.com/spring-cloud/spring-cloud-openfeign/blob/master/spring-cloud-openfeign-core/src/main/java/org/springframework/cloud/openfeign/FeignAutoConfiguration.java). The easiest would be to debug a remote call and to see which HTTP client is actually handling it.

If it is indeed executed by a supported HTTP client and it's still not working, could you enable the debug log level of the agent and send me the full agent log from startup of the application to one remote execution of feign?

Thanks,  
Felix

---

<div class="post-metadata">

**Author:** ![JohanRask](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johanrask/32/22713_2.png) [@JohanRask](https://discuss.elastic.co/u/JohanRask)\
**Post date:** [November 27, 2018, 11:03am UTC](https://discuss.elastic.co/t/distributed-tracing-http-calls-are-not-shown-in-span/158302/6 "2018-11-27T11:03:54Z")

</div>

I also tried with http-client (using it directly) without getting anything. My manual instrumented spans are displayed in kibana though. I can see in the apm-server log that nothing gets to the apm-server with auto instrumenting, only with manual instrumentation.

```
  <dependency>
      <groupId>org.apache.httpcomponents</groupId>
      <artifactId>httpclient</artifactId>
      <version>4.5.6</version>
    </dependency>

CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpGet = new HttpGet("http://google.se");
CloseableHttpResponse response = httpclient.execute(httpGet);
System.out.println(response.getStatusLine());

```

Thanks /Johan

---

<div class="post-metadata">

**Author:** ![jaks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaks/32/38097_2.png) [@jaks](https://discuss.elastic.co/u/jaks)\
**Post date:** [November 27, 2018, 2:08pm UTC](https://discuss.elastic.co/t/distributed-tracing-http-calls-are-not-shown-in-span/158302/7 "2018-11-27T14:08:05Z")

</div>

I have debugged the client and found that it is using ApacheHttpClient. Here is the screenshot,

 ![32%20PM](https://us1.discourse-cdn.com/elastic/original/3X/a/5/a5b8aa46744ed68a7313809563951a1cd46f2e21.png)

Here are the debug [logs](https://github.com/legendjaks/config-repo/blob/master/req.log)

Kindly let me know if I am doing something wrong.

---

<div class="post-metadata">

**Author:** ![jaks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaks/32/38097_2.png) [@jaks](https://discuss.elastic.co/u/jaks)\
**Post date:** [November 27, 2018, 2:10pm UTC](https://discuss.elastic.co/t/distributed-tracing-http-calls-are-not-shown-in-span/158302/8 "2018-11-27T14:10:31Z")

</div>

@JohanRask I didnt get your response completely. Are you getting spans for the above code?

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [November 27, 2018, 2:11pm UTC](https://discuss.elastic.co/t/distributed-tracing-http-calls-are-not-shown-in-span/158302/9 "2018-11-27T14:11:31Z")

</div>

Note that spans will only be created when there is an enclosing transaction for that. Could that be a problem in your case? I'll have a look at the logs in a bit.

---

<div class="post-metadata">

**Author:** ![JohanRask](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johanrask/32/22713_2.png) [@JohanRask](https://discuss.elastic.co/u/JohanRask)\
**Post date:** [November 27, 2018, 2:12pm UTC](https://discuss.elastic.co/t/distributed-tracing-http-calls-are-not-shown-in-span/158302/10 "2018-11-27T14:12:53Z")

</div>

Nope, that gives me nothing.

I have to code the instrumentation to get something into APM Server

`Transaction transaction = ElasticApm.startTransaction();`

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [November 27, 2018, 2:20pm UTC](https://discuss.elastic.co/t/distributed-tracing-http-calls-are-not-shown-in-span/158302/11 "2018-11-27T14:20:14Z")

</div>

Oh, I know what the problem is. The Apache HttpClient instrumentation is still marked as `incubating` by mistake! You can enable it by setting `disable_instrumentations` to `none`.

Sorry for that. Please tell me if that helped.

---

<div class="post-metadata">

**Author:** ![jaks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaks/32/38097_2.png) [@jaks](https://discuss.elastic.co/u/jaks)\
**Post date:** [November 27, 2018, 4:08pm UTC](https://discuss.elastic.co/t/distributed-tracing-http-calls-are-not-shown-in-span/158302/12 "2018-11-27T16:08:26Z")

</div>

Thanks @felixbarny. After adding `compile("com.netflix.feign:feign-httpclient:8.18.0")` and setting `disable_instrumentations` to `none`, spans are getting added for HTTP calls. 👍

---

<div class="post-metadata">

**Author:** ![jaks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaks/32/38097_2.png) [@jaks](https://discuss.elastic.co/u/jaks)\
**Post date:** [November 30, 2018, 3:11am UTC](https://discuss.elastic.co/t/distributed-tracing-http-calls-are-not-shown-in-span/158302/13 "2018-11-30T03:11:46Z")

</div>

There is an [issue](https://github.com/spring-cloud/spring-cloud-netflix/issues/1731) with `com.netflix.feign:feign-httpclient:8.18.0`, so use `io.github.openfeign:feign-httpclient:9.5.1`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 20, 2018, 11:11pm UTC](https://discuss.elastic.co/t/distributed-tracing-http-calls-are-not-shown-in-span/158302/14 "2018-12-20T23:11:56Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.