How to define span.destination.service.resource witn OpenTelemetry integration?

[davidpocar]

Hello,

I am using Open Telemetry integration with APM and am unable to define span.destination.service.resource.

I understand there is some mapping going on, as stated here: apm/specs/agents/tracing-api-otel.md at main · elastic/apm · GitHub. But even though the key is mentioned, I did not see anything related to it.

With db.system set to "mysql" and db.name set to "arst", the following keys are created:

• span.db.instance: arst
• span.db.type: mysql
• span.destination.service.name: mysql
• span.destination.service.resource: mysql
• span.destination.service.type: db

But I could not find how to change the resource. We have multiple instances we would like to see on a service map, so the desired values would be eg:

• span.db.instance: arst
• span.db.type: mysql
• span.destination.service.name: mysql
• span.destination.service.resource: mysql/arst
• span.destination.service.type: db

Is it not possible? Because I was able to change the resource for AMQP span after adding messaging.system and messaging.destination:

Thank you

[davidpocar]

Maybe to clarify.

My goal is to have consistent data across multiple services. This would also help in having more helpful service map that would show what service is using what database instance.

Some of our services use APM Agent which is creating mysql spans with resource defined like so:

Using OpenTelemetry, it is possible to do so with messaging services by sending messaging.destination, I was hoping it would be also possible for databases.

I can provide what is going to our exporter if that would help.

---

I would also like to apologize since I accidentally deleted the thread for a few seconds. I missclicked the delete button while trying to find an edit button to fix the typo in the title.

[Sylvain_Juge]

Hi !

There are two "intake protocols":

• the Elastic agents protocol, which is used by all our APM agents
• OTLP for OpenTelemetry data

Depending on which agent you are using there are different options for that:

• using the APM agent Tracing API to set the attributes, but that requires to execute it in within the database instrumentation
• using the APM agent and an ingest pipeline on the server side: Parse data using ingest pipelines | Elastic Observability [8.12] | Elastic that will require you to modify both the traces and some metrics to keep them consistent.
• using an opentelemetry collector (or an extension of the Otel agent) that retransforms the OTLP data using the database semantic conventions.

The third option is probably the simplest in your case, as the final mapping that happens in apm-server to transform the OTLP data in Elasticsearch documents would be applied consistently (so the metrics derived from traces are consistent without much effort).

[davidpocar]

Thanks for the reply!

We are using OTLP and these are the semantic conventions attributes that I have set in testing:

"attributes": {
  "code.function": "exec",
  "code.namespace": "PDO",
  "db.system": "mysql",
  "db.name": "some-instance-name",
  "db.statement": "SET time_zone = 'Europe\/Prague'",
  "db.connection_string": "mysql:\/\/user:password@localhost:3306\/database", // just to test
  "db.user": "user",
  "db.instance.id": "some-instance-id",
  "net.peer.name": "netpeername",
  "net.peer.port": 3306,
  "net.sock.family": "ipv4",
  "db.sql.table": "some-table",
  "db.operation": "SET"
},

In Elastic APM UI, these attributes are transformed like so:

According to this doc it looks like I am not able to add anything else concerning database span.
The span.destination.service.resource is still "mysql" even though I would expect it to be "mysql/some-instance-name".

We have a service that uses APM Elastic Agent in which the span resource is set to "mysql/production" and type is still "mysql". This is the state I would like to achieve with Open Telemetry integration. Is it even possible to edit span.destination.service.resource?

[Sylvain_Juge]

First, sorry for the late reply here.

As you are using an OTel agent the OTLP protocol is used and there is some translation happening in the apm-server side, so maybe there is something wrong there. I'll try to reproduce it on my side and keep you posted on this.

Which version of the stack (ES, apm-server, Kibana, ...) are you using here ?

[davidpocar]

@Sylvain_Juge

Which version of the stack (ES, apm-server, Kibana, ...) are you using here ?

Hello, thanks for the reply.
We are using Elastic Cloud 8.12.1.

[baoletrg (Bao Le Trong)]

Hi @Sylvain_Juge , any updates on this issue?
We are trying to switch from apm agent to otel agent, and this is blocking us. All the services are pointing to the same oracle dependency.

I have try to update the span.destination.service.resource to oracle/{{{db.instance.id}}}, this only works when seeing in the span details, but nothing new when showing in overall dependecies and service map.

Another thing is the external http span name, why only named GET or POST

[Sylvain_Juge]

The problem here is that the dependencies view is built on metrics that are derived from the tracing data before it gets written, so an ingest pipeline on the tracing data will not cover it.

You can identify such metric documents by querying data_stream.dataset:"apm.service_destination.1m", and unfortunately the original span attributes like db.instance are not preserved for those documents, so you can't create an ingest pipeline to cover those.

I've opened a draft PR to fix this directly into APM server for the database spans to limit potential impact: fix otlp database span resource by SylvainJuge · Pull Request #241 · elastic/apm-data · GitHub however I do not have an ETA when and if this will get merged so that's more a medium/long term solution.

I think that we might get a work-around to this limitation by using a custom processor (at agent or at collector level) and two ingest pipelines:

• with a custom processor (before the APM server), add a custom attribute to the DB spans that is outside of the database semantic conventions and copy the value currently set to db.name attribute, for example to custom_db_instance, this will get copied to a label accessible in the labels.custom_db_instance field in the trace documents and also replicated to the derived metrics documents.
• create an traces-apm@custom ingest pipeline where you can just discard this extra redundant custom_db_instance attribute and set the expected value for span.destination.service.resource for the relevant database spans.
• create an metrics-apm.service_destination.<metricset.interval>@custom for each relevant interval (by default it's 1m and 10m) that will modify the service destination metrics by reading the value from custom_db_instance label, which can then be discarded.

That's a bit sketchy but I think that it should work. This relies on the fact that the attributes that are part of the semantic conventions like db.* are mapped to ECS fields and are not kept to avoid redundancy, but the custom attributes that are not mapped are just copied as-is to the span/metrics labels, which allows us to use those for "transport" here.

[baoletrg (Bao Le Trong)]

Hi @Sylvain_Juge thanks for your help, I can finally customize the oracle dependency name. In my case, it is easier than you think, I update the field span.destination.service.resource to {{{service.target.type}}}/{{{service.target.name}}.
And create 4 custom ingest pipelines:
traces-apm@custom
metrics-apm.service_destination.1m@custom
metrics-apm.service_destination.10m@custom
metrics-apm.service_destination.60m@custom

Thanks a lot for your help.

[davidpocar]

Thank you for the detailed answer and also the draft. Hopefully it gets merged some day.

We are currently working on a collector so I hope to implement the provided workaround. If not, we will wait for the PR to be merged (hopefully).

Nonetheless, my question was answered, so I am marking this as solved.