How to visualize or count same Counter from different pods

Elastic Observability Metrics
1

We are using OpenTelemetry to collect custom metrics into Elastic.

We have a simple counter metrics that on each of our endpoint call, increase +1. (basically we want to count how many times an endpoint called in X period of time)

Until here everything looks good, now we have many pods of the same service that may increase this counter for our endpoints. So each pods has its own value for this counter. Also each time a pod restart, or when a new pods created this counter reset to 0 for that pod . (This is happen by OpenElemetry collector by design I assume, and we don't want to reset it to 0 ).

Any suggestion how we can visualize or measure this counter though elastic visualization? As an example we want to want to know in the beginning and end of a month, what was the value for that metric? ( Considering various pods had different values and various times the value reset to 0 )

2

Hi @Mazdak_Mansouri, Welcome to the Elastic Community.

  1. May I know which OTel collector you are using or is it custom metrics written by you?
  2. If it is custom, How you Elasticsearch documents looks like?
  3. When you say you updating the counter -
    a. Is it updating one document ?
    b. Or you are injecting one record on every endpoint hit ?

Thanks

3

Hi Mazdak, when creating the counter in the SDK, use 'delta temporality' rather than 'cumulative temporality'. Then in Kibana, use Lens with Sum function. While we can do the delta in Kibana, it gets more complicated when counter=0 on pod reset. The 'sum', of course, is calculated over the time frame the Lens is run over.

4

Sorry for delay,
Thank you both for replies.
The issue was fixed with using delta temporality and we could get the expected result.

Thank you very much.