Python Agent how to track various counters/values evolution over time?

Elastic Observability APM
1

I'm really new to APM & Kibana, but ok with Python & ElasticSearch, so forgive me my question seems stupid. Before I had Graphite and it was quite easy to do custom tracking.

I'm looking to track 3 simple custom metrics and their evolution over time.

  1. Counter and it's value. For example queue_size: 23 and send it by any of the workers. What happens when different workers send different values? (because of the time, the value might increase/decrease rapidly).
    I do have 20 names of queues to track. Should I put all under a service_name or should I use labels?
    Before I used:
    self._graphite.gauge("service.queuesize", 3322)
    No idea what to have here:
    ....

  2. Time spent within a method. I saw here it's possible to have a context manager.
    Before I had:
    with self._graphite.timer("service.action")
    Will become
    with elasticapm.capture_span('service.action')

  3. Number of requests. (only count no other tracking)
    Before I had
    self._graphite.incr("service.incoming_requests")
    Is this correct?
    client.begin_transaction('processors')
    client.end_transaction('processors')
    ...

THanks a lot!

2

Hi @AlexRa, welcome to the forum!

Counter and it's value. For example queue_size: 23 and send it by any of the workers

How are you tracking the queue size? Is this something periodic, i.e. every 30 seconds the worker will query and report the queue size? Or are the changes associated with each request?

Custom periodic metrics aren't currently supported by the Python agent, but may be in the future. If it's a number that you can attach to a transaction, then you could label the transaction with the value: API reference | APM Python Agent Reference [6.x] | Elastic

For what it's worth, I'd use a separate "queue_name" label so you can perform aggregations over it like "top 5 queues by size".

Time spent within a method. I saw here it's possible to have a context manager.

That's right, assuming that a transaction is in progress (i.e. begin_transaction was called) within the active context.

Note that capture_span takes a name, type, and optionally subtype and action. For example, database queries have the type db, a subytpe like mysql or postgresql, and action query. You may want to do something similar if you have multiple span types, so you can more easily search and aggregate them.

Number of requests. (only count no other tracking)

Yes, begin_transaction and end_transaction are the right calls to make for this. You'll be able to query the request rate and latency distribution as a result.

How do the requests come in? If it's a Django or Flask-based application, then bear in mind that the agent comes with support for those out of the box, and you don't need to be calling the begin/end_transaction methods yourself.

3

thanks for taking time to reply.
While the most important thing now is to have support for custom metrics, or at least find a solution to get it done for now.

Some more documentation would be great, as it's quite limited to only a few common use cases. For now I can't really see the benefit for switching to it, after spending like 1 week trying to figure out things on my own.

Even the python agent has lots of issues.

4

There is an undocumented API which you can use to report your own custom metrics, but please understand that this is unsupported at the moment, and its API may change. It would be useful for us to know your thoughts on making this an officially supported API - whether it does what you need already, or what's missing.

The elasticapm.Client has a _metrics attribute, holding a metrics registry. It is possible to register a subclass of MetricsSet by calling client._metrics.register(class_path).

So for example if you have a package called "custom_metrics", then you create a subclass of MetricsSet in that package called Set, and then call client._metrics.register("custom_metrics.Set") to register it. Set should have a before_collect method which will be periodically invoked in order to create/update metrics.

$PYTHONPATH/custom_metrics.py:

from elasticapm.metrics.base_metrics import MetricsSet

class Set(MetricsSet):
    def __init__(self, registry):
        super(Set, self).__init__(registry)

    def before_collect(self):
        self.gauge("queue_size", queue_name="queue0").val = 123
        self.gauge("queue_size", queue_name="queue1").val = 456

If you can provide any more details on which use cases are missing from the docs, and which issues you're referring to, that would be helpful.

For the docs, I suppose extending the Instrumenting custom code docs with examples of starting/ending transactions might help?

5

There's also a setting that can be used to register metrics sets: https://github.com/elastic/apm-agent-python/blob/bca5828503bd42825bb9d8e338e3f9a990d27bfd/elasticapm/conf/init.py#L285-L291.

We haven't documented this on purpose, as the API isn't stable yet and may change without warning in upcoming releases.

6

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.