Any update for RMQ request/response tracing which uses Python pika client

Dixit · September 23, 2024, 5:54am

Dixit · September 23, 2024, 5:54am

Add instrumentation and distributed tracing for RabbitMQ (pika)

opened 10:46AM - 18 Dec 19 UTC

enhancement agent-python instrumentation

**Context** One common way to integrate services is using RabbitMQ as a message… broker. Currently, the Elastic APM agent only supports distributed tracing for HTTP calls. It would be of high value to add instrumentation and distributed tracing for message brokers and, in particular, [RabbitMQ]( https://www.rabbitmq.com/) - one of the most deployed open source message brokers. Not having this integration could be a show-stopper for users of competitor solutions ([New Relic already sports this integration](https://docs.newrelic.com/docs/agents/python-agent/getting-started/instrumented-python-packages#message-broker-clients)) when considering migrating to Elastic APM. **Solution** The ideal solution would be to instrument message publishing and include tracing data in message headers, interpreting those headers when consuming messages to continue the tracing. To support the maximum number of projects using RabbitMQ, I would recommend to start by adding support for [Pika](https://pypi.org/project/pika/) - a widely-used AMQP client library having support for both Python 2.7 and +3.4. **Alternatives** A flexible alternative to a library-specific integration could be to provide APIs to create and interpret a distributed tracing payload for use in custom instrumentation of any type of communication between services. New Relic, for example, does offer these APIs in [`create_distributed_trace_payload()`](https://docs.newrelic.com/docs/agents/python-agent/python-agent-api/createdistributedtracepayload-python-agent-api) and [`accept_distributed_trace_payload()`](https://docs.newrelic.com/docs/agents/python-agent/python-agent-api/acceptdistributedtracepayload-python-agent-api) I'm willing to contribute with code if this is of general interest. / Hampus

Dixit · September 23, 2024, 5:54am

github.com/elastic/apm

Add messaging fields to schema

opened 12:10PM - 10 Sep 19 UTC

closed 08:20AM - 08 Mar 21 UTC

eyalkoren

## Description of the issue Adding dedicated fields to schema to support messag…ing context. ## What we are voting on ### New types #### Background Message send/publish events can only be captured as spans if occurring within a traced transaction. Message consumption can be divided into two types: passive, where you would implement a listener that is called once a message is available, and active, where the queue is being polled (blocking or non-blocking). Passive consumption typically results in a Messaging transaction and is pretty straightforward to trace. Message polling can be done within a traced transaction, in which case it will result in a messaging span, or it can be the initiating event for a message handling flow. Capturing polling spans is also mostly straightforward. For polling-based transactions, our goal is to capture the message handling flow, which typically starts after the polling action exits, returning a message. This may be tricky if the handling flow is not implemented within a well defined API. The actual implementation for that may be different for different clients. Whenever it is not possible (or too risky), we may consider capturing the message polling action itself, but this should be decided for each case separately and should only be used as a last resort. In any case, we should not create a transaction based on polling APIs if the polling action did not result with a message (as opposed to polling spans, where we want to capture such as well). #### Proposed types - Transactions: `messaging` for `transaction.type` - Spans: `messaging` for `span.type` (both sends and receives-within-a-transaction), the name of the framework for `span.subtype` (eg `jms` or `kafka`) and the operation for `span.action` (eg `send`, `receive`, `poll`). ### New fields - **`context.message.queue.name`**: optional for `messaging` spans and transactions. Indexed as keyword. Wherever the broker terminology uses "topic", this field will contain the topic name. - **`context.message.body`**: similar to HTTP requests' `context.request.body`- only fill in messaging-related **transactions** (ie incoming messages creating a transaction) and not for outgoing messaging spans. - Rely on the existing `ELASTIC_APM_CAPTURE_BODY` agent config option. - Only capture UTF-8 encoded message bodies - Limit size to 10000 characters. If longer than this size, trim to 9999 and append with ellipsis - Storage: same Elasticsearch mapping as `context.request.body` - **`context.message.headers`**: similar to HTTP requests' `context.request.headers`- only fill in messaging-related **transactions**. - Rely on the existing `ELASTIC_APM_CAPTURE_HEADERS` agent config option. - Sanitize headers with keys configured through `ELASTIC_APM_SANITIZE_FIELD_NAMES` - Intake: key-value pairs, same like `context.request.headers` - Storage: same mapping as used for `context.request.headers`. - **`context.message.age.ms`**: a numeric field indicating the message's age in milliseconds. Relevant for transactions and `receive` spans that receive valid messages. There is no accurate definition as to how this is calculated. If the messaging framework provides a timestamp for the message- agents may use it. Otherwise, the sending agent can add a timestamp _indicated as milliseconds since epoch UTC_ to the message's metadata to be retrieved by the receiving agent. If a timestamp is not available- agents should omit this field. Clock skews between agents are ignored, unless the calculated age (receive-timestamp minus send-timestamp) is negative, in which case the agent should report 0 for this field. ## Vote @elastic/apm-agent-devs and @elastic/apm-server: tick the N/A box if irrelevant for you. Otherwise, link to the relevant issue or PR this and tick "Yes". | Component | Yes | N/A | Link to component issue | -------------|:----:|:----:|:-------------------:| | .NET |<ul><li> [ ] </li></ul>|<ul><li> [ ] </li></ul>| | Go |<ul><li> [x] </li></ul>|<ul><li> [ ] </li></ul>| | Java |<ul><li> [x] </li></ul>|<ul><li> [ ] </li></ul>| https://github.com/elastic/apm-agent-java/pull/664 | Node.js |<ul><li> [ ] </li></ul>|<ul><li> [ ] </li></ul>| | Python |<ul><li> [ ] </li></ul>|<ul><li> [ ] </li></ul>| | Ruby |<ul><li> [ ] </li></ul>|<ul><li> [x] </li></ul>| | RUM |<ul><li> [ ] </li></ul>|<ul><li> [x] </li></ul>| | Server |<ul><li> [ ] </li></ul>|<ul><li> [ ] </li></ul>| https://github.com/elastic/apm-server/issues/2697

Dixit · September 25, 2024, 11:18am

@basepi is there any agent update for RMQ other than custom implementation ?
Am having some confusion over custom implementation in RMQ tracing which is being called from HTTP.

Some code samples would help

Dixit · October 8, 2024, 5:35am

@basepi Please can you help with this rquest

basepi · October 17, 2024, 7:09pm

Unfortunately, I don't have the bandwidth to help with custom instrumentation. Apologies!

Dixit · October 18, 2024, 6:43am

thanks @basepi some sample RMQ - Produce & Consume methods (challenge is in here) in Elastic would help, would try out.