APM integration with Istio service mesh using w3c tracecontext

[Agzem]

Kibana version: 7.10.2

Elasticsearch version: 7.10.2

APM Server version: 7.10.2

APM Agent language and version: java - 1.20.0

Browser version: Chrome, latest

Original install method (e.g. download page, yum, deb, from source, etc.) and version: ECK stack and beats installed with elastic operator, logstash with custom helm.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

I'm currently trying to integrate elastic APM server and agent binding to my spring cloud application with Istio inside my cluster with w3c traceparent methods. the flow is listed below.

spring : using spring cloud sleuth 3 with trace propagation as w3c
spring communication : spring open feign
apm agent : using default mode which is w3c
istio : using opencensus agent bridge to propagate w3c the trace from spring but not exporting anything

trace flow from above is working and I'm able to track all the traces inside my filebeat-* and apm-* index fine, but when I go to my APM dashboard(the APM UI in observability) any service to service communication will be labeled with External and show only IP to that service like image below

and if I go into that span detail it will be like this image

Steps to reproduce:

1. install istio and enabled trace propagation

 meshConfig:
    enableTracing: true
 proxyConfig:
    tracing: 
        openCensusAgent:
            context: ["W3C_TRACE_CONTEXT"]

2. install elastic stack+filebeat+apm server with eck and logstash with helm
3. configure spring cloud application binding with java apm agent with init container method
4. perform http request on service to service communication inside k8s
5. verify that service that getting called from service-a is display as ip instead of apm tracked service

how do I fix this apm display as external ip instead of actual service name that get calls? because service traffic will get routed by envoy proxy sidecar.

[Sylvain_Juge]

Hi @Agzem ,

Here, the external just next to the http indicates that the application made a request to another service through HTTP, there is nothing related to the IP address being internal or external from network point of view.

APM agent always has the same perspective as the process it's been deployed on, thus from the monitored application point of view, the only visible service IP is the 9.9.9.9 which is routed somewhere else, but the application can't be aware of that.

[Agzem]

Thank you for your reply,
If I understand correctly, that mean, I have to configured my istio envoy sidecar to also export its metric to APM server to make APM aware of that service in order to make APM dashboard display normally? because the service IP 9.9.9.9 display here is actually IP of service-b that also spring application with APM agent sitting in the same namespace as service-a

[Sylvain_Juge]

Hi Agzem,

I don't know how istio works, thus I'm not sure how I could help on your specific case.

What I meant in my previous message is that APM agent is not aware of your infrastructure and the network topology, if the application communicates to a specific IP address like 9.9.9.9 and that one happens to be routed somewhere else with another different IP or name the agent won't be able to know about it.

Here maybe using DNS names (or simple entries in /etc/hosts) to make sure that the logical names are being used consistently would allow to work-around.

For example, having 9.9.9.9 dbserver in /etc/hosts of your application host would allow the application to use dbserver name at application level without relying on the routed IP address 9.9.9.9.