APM Custom Counters?

Elastic Observability APM
1

Hi,

If I would like to have some custom counters (let's say numbers), how would that be represented in the context of Elastic APM?

Unless I overlooked the REST API, seems like there is no way to add custom fields. The closest I can think of is to use Tags, but the value field is String type though. I would like to graph the counters, so String isn't good for me.

Any suggestions on how to do that?

Thanks,
Ed

2

Hi Ed,

Can you please give an example of what sort of counters you want to report? We're looking into providing an API for applications to report metrics, but I'm not sure if it's what you're asking for.

Cheers,
Andrew

3

Hi Andrew,

Thanks for the quick reply.

Some examples of counters in mind are:

  • Number of get/post/put requests
  • Number of files currently opened
  • Number of running threads
  • Number of misses from cache

We would like generate graphs from these counters too. For instance, how many files open in the last hour, etc. Basically any custom domain/application specific metrics that didn't fit into the "span" model. To my understanding, the Span model is more suitable for timing things.

We are heavy on C# and acknowledged that there isn't a C# agent yet. We are considering to make use of the REST API from the APM Server to gather our performance metrics (timing data and counters).

Custom counters are important to us. Would you please clarify whether we can do custom counters with the current RESP APIs? If so, an example would be great. If not, any other workaround or suggestions?

Thanks!
Ed

4

Number of get/post/put requests
Number of files currently opened
Number of running threads
Number of misses from cache

The first one should be answerable using HTTP request transaction data, since the schema includes the HTTP method. Of course, in your case that assumes a C# agent exists, which as you know is not currently the case.

As for the rest: we're working on it, but there's currently no API for transmitting application metrics.

We would like generate graphs from these counters too. For instance, how many files open in the last hour, etc. Basically any custom domain/application specific metrics that didn't fit into the "span" model. To my understanding, the Span model is more suitable for timing things.

Graphs will definitely be considered important. The current thinking is that these metrics (custom counters, etc.) will be consumed using Kibana's Time Series Visual Builder.

We are heavy on C# and acknowledged that there isn't a C# agent yet. We are considering to make use of the REST API from the APM Server to gather our performance metrics (timing data and counters).

Custom counters are important to us. Would you please clarify whether we can do custom counters with the current RESP APIs? If so, an example would be great. If not, any other workaround or suggestions?

As mentioned above, currently there is no support for metrics beyond what you get from transaction/span data, but we'll be working on it. Custom/application-specific counters shouldn't be a problem.

Given that there's no API in APM for this yet, I'd suggest that in the interim, consider feeding the counters into Elasticsearch by some other means, e.g.:

5

Thanks again Andrew for the prompt reply. It's clear to me what's the current status of the APM APIs and what's in your roadmap regarding custom counters. Very glad that you guys will be working on this.

Looking at the pre-alpha Java agent, as a temporary workaround, do you think we can make use of the Tags for custom counters?

According to this (https://github.com/elastic/apm-agent-java/blob/master/docs/public-api.asciidoc#void-addtagstring-key-string-value), tags are indexed so that they are searchable and aggregatable.

For instance, we are considering tags like the following to represent a single counter:

  • counterId: numActiveThreads
  • numActiveThreads: 13

Reason to split up into 2 tags, instead of numActiveThreads: 13, is to help converting the string value to number (or adding new field) with a elasticsearch pipeline (since tag only supports string value).

Once we can ingest tag values as numbers then we can visualize them in Kibana. Do you think this will work? Do you see any other possible downside with this temporary solution?

Thanks!
Ed

6

Looking at the REST API more closely and realized the approach with using Tags for counters is probably not a good idea.

Even though I believe technically it will work but there are quite a few down sides:

  • The duration field is a required field, that means counters will have duration, which doesn't quite make sense
  • Given it has duration, it will "screw up" the default APM GUI

I have scrapped that idea now. Thanks again for your help Andrew.

Ed

7

Although it might not be the way to go in your case, I like the idea of having support for number tags. That's why I created https://github.com/elastic/apm-server/issues/828.

8

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.