Azure Monitor Metrics Integration Reporting interval

[s.buksa (Sintija Bukša)]

Hello,
In Azure Monitor Metrics integration official documentation under "Data stream specific configuration notes" mentioned that "Period (reporting interval) for Metrics will have a timegrain of 5 minutes, so the 'Period' configuration option for monitor should have a value of '300s'"
If I have one resource which supports metrics with different timegrains, for example, one metrics supports only PT5M, another one PT1M and some supports only PT1H or even PT1D - does it change something in ingested data values if several metrics with different timegrains (PT1M, PT5M, PT1H etc.) are included in one configuration for which period at the beginning of configuration is provided as "60s" instead of "300s"? What is the logic to follow to if metrics supports different timegrains and what would be proper period configuration? Could not find detailed quidance in such case.

For example:

period: 60s
client_id: xxx
client_secret: xxx
tenant_id: xxx
subscription_id: xxx
refresh_list_interval: 60s
resources:
  - resource_query: "resourceType eq 'Microsoft.DocumentDb/databaseAccounts'"
    metrics:
      - name: 
          - CreateAccount
          - DeleteAccount
        timegrain: PT5M
        namespace: "Microsoft.DocumentDb/databaseAccounts"
        ignore_unsupported: true
      - name:
          - MongoRequests
        timegrain: PT1M
        namespace: "Microsoft.DocumentDb/databaseAccounts"
        ignore_unsupported: true
        dimensions:
          - name: DatabaseName
            value: '*'
      - name:
          - ServiceAvailability
        timegrain: PT1H
        namespace: "Microsoft.DocumentDb/databaseAccounts"
        ignore_unsupported: true

[s.buksa (Sintija Bukša)]

@stephenb is there anyone from Elastic team who could help me to figure it out?
Thanks.

[stephenb (Stephen Brown)]

Hi @s.buksa
I'm not familiar with this at all. Can you at least link to the documentation you're referring to?

I guess I'd try different settings and then look and discover to see what the actual documents are doing. That's usually my way of debugging these things when I don't understand them.

[s.buksa (Sintija Bukša)]

Hi, thank you for your reply!
Here is the link to official documentation: https://docs.elastic.co/en/integrations/azure_metrics/monitor
If you could suggest me someone familiar/responsible for it to consult with, it would help a lot!

[stephenb (Stephen Brown)]

I will see if I can poke around and find someone

Here is the actual code

[s.buksa (Sintija Bukša)]

Thank you!

Will look forward to hearing.

[stephenb (Stephen Brown)]

No promises...

I will ask why you think you should define timegrain that is not a parameter as far as I can tell ... Only period which be in see you should simply set Period as a multiple of 300s

And yes you should be able to define multiple metrics with individual configurations as far as I can tell

[s.buksa (Sintija Bukša)]

Basically yes, you understood me correctly.

If additional comment from my side helps, then :
What I understand from source code is that metrics included in configuration contains only PT1M and PT5M. As per my understanding reporting interval provided there as 300s (PT5M) is related to highest timegrain mentioned in configuration.
However, if the timegrain in configuration is PT1H or even PT1D - would it mean that reporting interval should be equivalent to highest timegrain in configuration?
Or it could be any, even 60s (desired in my case) regardless of how many different timegrains are mentioned as supported by particular metrics.

[zmoog (Maurizio Branca)]

The current version of the Azure Monitor integration works best when the period and the timegrain are aligned (PT5M with a 300s period). That's why most Azure metrics integrations try to keep one timegrain when possible ^[1].

However, some integrations have multiple timegrains (PT1M and PT5M). When timegrains < period, the metricset may collecte values less frequently. Instead, when timegrains > period, the metricset may collect the same value more than once.

To avoid these problems, users can set up the generic Azure Monitor integration to use aligned timegrain and period, like AWS integration does.

We are also working on an enhanced version of Azure Metrics that avoids collecting metrics more than once for the timegrains > period. This change will probably ship on 8.12.

With the updated Azure Monitor, users can collect metrics without duplicates on integrations with PT1M, PT5M, and PT1H timegrains by setting the period to 60s.

---

1. The AWS integration applies this rule strictly, and "timegrain" and period are always perfectly aligned. ↩︎

[s.buksa (Sintija Bukša)]

Hello,
Thank you for provided information. Upgraded to elastic agent version 8.11.2 (integration version 1.4.1) as soon as it became available. Can see a very good improvement related to ingested data. However, two things do not leave my mind and cannot understand that particular behavior.

1. Two metrics for Microsoft.DocumentDb/databaseAccounts are not being picked up from Azure Monitor when using Azure Monitor Metrics integratio, i.e., ProvisionedThroughput and NormalizedRUConsumption. As soon as I switch back to previous version 8.11.1 metrics fields appear in elasticsearch. Is there any way how can I make sure that it really is because version/integration upgrade? There is even no any error message - like missing metrics has never been touched. And happens only with latest version.

2. When I set period to 60s, metrics that are listed in configuration with PT5M are not grabbed to. Fields itself appears in elasticsearch but do not contain any information although they should if we compare to metrics in Azure. For metrics that has PT1M everything looks fine.

3. When I set periof to 300s, metrics that are listed in configuration with PT5M are grabbed and appears in Elasticsearch. But again, it leads to ingested metrics latency that share PT1M timegrain and it is not a good choise to go for if there are crucial metrics that needs near real time monitoring without significan latency.

4. Seems to be working just fine, if I create two separate sectiond with different id where each section contains only metrics supporting specific timegrain. And here again two of all metrics are missing - previously mentioned ProvisionedThroughput and NormalizedRUConsumption.