Azure Module in Metricbeat : Monitor Metricset

wadhah · June 11, 2020, 4:25pm

Hello,

I want to make efficient use of "Monitor" metricset in "Azure" module on metricbeat side.
I am looking to get relevant information regarding disk usage, memory availability of my Azure VMs.
The issue is that I am getting almost the same data as "Compute_vm" metricset.
I was wondering, what exact Config I need to do on Azure side.
An other issue that I encountered, is that I am using "include_fields" processor to filter on the data I am getting in elasticsearch. I have put "azure.monitor.metrics" pattern as described in the documentation, however I was unable to fetch any kind of data.
When I got rid of the processor, I received data in elasticsearch and I discoverd that the pattern is "azure.metrics" and not "azure.monitor.metrics", so I was wondering if this is some kind of a bug.

Kind Regards

jsoriano · June 15, 2020, 4:00pm

Hi @wadhah,

It is very possible that you see similar metrics when using the compute_vm and when using the monitor metricset to query information about Azure VMs. Under the hood they both use Azure Monitor to retrieve this information, the monitor metricset in a more generic way, and compute_vm more focused on VMs.

Is there any metric you are missing in the compute_vm metricset? Consider opening an enhancement request if you think that something could be improved

This is expected. Metrics collected from Azure in a generic way using the monitor metricset are stored in azure.metrics. You can find an example event in the docs: Azure monitor metricset | Metricbeat Reference [7.7] | Elastic

wadhah · June 15, 2020, 4:25pm

@jsoriano Thank you so much for the response.
The main metrics I am trying to track are CPU, memory and disk_usage.
Thanks to "monitor" or "compute_vm" I am able to retrieve metrics related to CPU_usage, however for the rest, I find myself unable to fetch data.
I referred to :

which takes place in metricbeat configuration. So, theoretically, it is possible to retrieve at least data regarding VM memory availability.
For the record, I enabled "VM guest metrics" on Azure side as stated in the documentation, still it didn't work.
That's why I am wondering if I need to take more action on Azure side (if so, what are these actions).
Thanks in advance

jsoriano · June 15, 2020, 6:01pm

In the screenshot you are sharing I can see that you have at least some metrics about available memory, are these the metrics you are looking for?

Regarding memory metrics and others like the disk usage ones, take into account that Azure Monitor monitors virtual machines from "outside", from the host side, so it doesn't have a fine-grained view of what the guest operating system is doing with the assigned resources. To have this kind of metrics you probably need to deploy a monitoring agent like Metricbeat inside the virtual machine.

wadhah · June 16, 2020, 8:40am

Thanks again @jsoriano for your quick interaction.
In fact the screenshot presented above is taken from the official Metricbeat documentation.
So, to respond on your question, yes those are the kind of metrics I am seeking, however, regarding available memory I am not able to fetch it, neither through "compute_vm", nor "monitor" metricsets.
Therefore, I agree, I think then we would have to deploy metricbeat agents on the virtual machines.

jsoriano · June 16, 2020, 4:47pm

Oh ok, are you using these dashboards with your data? do they show something? This graph uses the azure.compute_vm.memory_available_bytes.avg field.

wadhah · June 17, 2020, 9:38am

Thanks again @jsoriano for your quick responses.
Yes I am using the dashboards with my data. All the visualizations work just fine.
However, the "VM Available Memory " shows no data. Furthermore, "azure.compute_vm.memory_available_bytes.avg" was never been a part of the fields generated by neither "compute_vm" nor "monitor" metricsets.
I am using a processor "include_fields" in my config and I am calling "azure.compute_vm" and "azure.metrics" fields (so I am supposed to get all possible fields, but I have never got the field that you've mentioned above). I even included "azure.compute_vm.memory_available_bytes.avg" explicitly in the list of fields, but as well, nothing changed.

MarianaD · June 17, 2020, 5:47pm

hi @wadhah,

What is the os you are running on those vm's?
Also , if you check inside the Azure portal in the Monitor/Metrics area of any of those vm's, do you see the guest namespace and the memory metrics? If so, are there any values for those metrics?
If it's a yes to all of that, can you provide us with the current configuration (please remove any sensitive information before) .

wadhah · June 18, 2020, 10:26am

Hello @MarianaD, thank you for the interaction.
Actually, I have both Windows and Linux VM.
Regarding your 2nd question, yes, on Azure portal, I am able to see the "Guest" namespace, and have an overview of the memory metrics.
And for the record, the ELK stack is running on Azure Kubernetes Cluster and Metricbeat is deployed as a daemonset.
Here's my modules configuration:

- module: azure

     metricsets:

         - monitor

      enabled: true

      period: 300s

      client_id: '${AZURE_CLIENT_ID}'

      client_secret: '${AZURE_CLIENT_SECRET}'

      tenant_id: '${AZURE_TENANT_ID}'

      subscription_id: '${AZURE_SUBSCRIPTION_ID}'

      resources:

        - resource_query: "resourceType eq 'Microsoft.Compute/virtualMachines'"

          metrics:

          - name: ["*"]

            namespace: "Microsoft.Compute/virtualMachines"

Just for the record, I think that I have to use an other namespace to get the "Guest" metrics.
However, when I make the API call to list the namespaces for my Linux VMs, I only get the above namespace.

MarianaD · June 18, 2020, 12:20pm

hi @wadhah, a few points here:

guest metrics are not available on linux vm's at the moment due to the fact that the Azure monitor data sink is only supported on Windows VMs , they do have a workaround found here https://github.com/MicrosoftDocs/azure-docs/issues/28457#issuecomment-524081288
This is why the guest namespace is not found for the linux machines.
with the configuration above you will not be able to retrieve any guest metrics because as you have also mentioned the namespace selected does not contain those metrics.
Have you tried using the compute_vm metricset and test it on a windows vm only just to see if the memory metrics are coming through?

- module: azure
  metricsets:
  - compute_vm
  enabled: true
  period: 300s
  client_id: '${AZURE_CLIENT_ID:""}'
  client_secret: '${AZURE_CLIENT_SECRET:""}'
  tenant_id: '${AZURE_TENANT_ID:""}'
  subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'
  refresh_list_interval: 600s
  resources:
  - resource_id: "{resource_id}"

Can you enable debug logging and check if those metrics are found in the events sent?

wadhah · June 18, 2020, 1:18pm

Thank you again @MarianaD for the update.
Regarding the 1st point, I was wondering if by workaround you mean, installing telegraf agents on the VMs? If so, yes I have already tried that solution, it works like a charm, but the disadvantage here, is that it will get complicated once we're dealing with 100s of VMs.
As for the 2nd point, honestly, I didn't try it with windows VMs yet. I will give it a try and get back to you with feedback.

wadhah · June 30, 2020, 2:32pm

Hello again.
We faced the same issue, nothing has really changed

MarianaD · July 1, 2020, 1:48pm

hi @wadhah, can you go over the steps you have taken?
For example can you single out a windows vm which has the guest namespace enable in Azure monitor.
Then in Metricbeat can you configure the azure.yml similar to this:

- module: azure
  metricsets:
  - compute_vm
  enabled: true
  period: 300s
  client_id: '${AZURE_CLIENT_ID:""}'
  client_secret: '${AZURE_CLIENT_SECRET:""}'
  tenant_id: '${AZURE_TENANT_ID:""}'
  subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'
  refresh_list_interval: 600s
  resources:
  - resource_id: "{resource_id}"

and provide the resource id from that specific windows vm.
After that can you execute the query:

get metricbeat-8.0.0*/_search
{
  "query": {
    "term": {
      "metricset.name": {
        "value": "compute_vm"
      }
    }
  },
  "size": 100,
  "sort": [
    {
      "@timestamp": {
        "order": "desc"
      }
    }
  ]
}

If no events are returned, can you check the Metricbeat logs (enable debug logging) in order to see if any events are sent, you can also check the elasticsearch logs for any parsing mapping exceptions as well.

wadhah · July 1, 2020, 2:07pm

Thanks @MarianaD for your quick updates as usual.
It works as long as I am specifying the namespace in "monitor" metricset.

I just have an other question if you please, that is not related directly to this, but :
I am using Kubernetes module to get relevant data regarding the pods performance.
Everything is working as expected.
The metricsets I am relying on are pod, state_pod, state_statefulset .....
I was wondering if there's a way to get the type/name of the deployment in the same document generated by the metricset pod :slight_smile.

So let's say for example, thanks to "pod" metricset I get these fields in the same document:

"kubernetes.pod.name" : "pod-klk23"
"kubernetes.pod.memory.available.bytes" : 900Mb

-> Is there a way to include something like "kuberntes.pod.deployment.type" (which takes as value statefulset or deployment or daemonset) or "kubernetes.pod.deployment.name" (which takes for example "kibana" as a value) ?

-> I thought about using processors, but I didn't find something that could help.

system · July 29, 2020, 4:08pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.