Metricbeat Windows module claims "failed to query" but still has data

[amorrow (Tony Morrow)]

Hi. I use Metricbeat to collect data from some Windows servers using perfmon. I'm in the process of upgrading Metricbeat from 7.16.0 to 8.13.0 and ran into a weird issue where the output has an "error.message" field saying "failed to query=", but the document has the value the item that it says it failed on. I'm not sure why that is occurring and I don't get the error messages with v7.16. Has anyone ran into this issue or knows why it could be occurring. Thanks.

My process for upgrading was the following:

1. Stop the metricbeat service
2. Use the powershell script to uninstall the metricbeat service
3. Rename the existing metricbeat folder
4. Download and extract metricbeat 8.13.0
5. Copy my metricbeat.yml file from the old installation folder to folder for the new installation
6. Copy my windows.yml settings from the old installation to the modules.d folder in the new installation
7. Use the powershell script to install the new metricbeat service
8. Start the metric beat service.

This is what my windows.yml file looks like:

- module: windows
  metricsets: [perfmon]
  period: 15s
  perfmon.ignore_non_existent_counters: true
  perfmon.group_measurements_by_instance: true
  perfmon.queries:
  #System Performance
    - object: "Processor"
      instance: "*"
      counters:
        - name: "% Processor Time"
          field: processor_time_pct
    - object: "Memory"
      counters:
        - name: "Available MBytes"
          field: memory_available_mbytes
    - object: "Paging File"
      instance: "_Total"
      counters:
        - name: "% Usage"
          field: pagefile_usage_pct        
    - object: "PhysicalDisk"
      instance: "*"
      counters:
        - name: "Disk Writes/sec"
          field: physical_disk_write_per_sec
        - name: "Disk Reads/sec"
          field: physical_disk_read_per_sec
        - name: "Avg. Disk sec/Write"
          field: physical_disk_write_avg_per_sec
        - name: "Avg. Disk sec/Read"
          field: physical_disk_read_avg_per_sec
    #SQL Server
    - object: "SQLServer:General Statistics"
      counters:
        - name: "User Connections"
          field: sqlserver_general_staistics_user_connections
    - object: "SQLServer:Databases"
      instance: "*"
      counters:
        - name: "Transactions/sec"
          field: sqlserver_databases_tranactions_per_sec
    - object: "SQLServer:Buffer Manager"
      counters:
        - name: "Page life expectancy"
          field: sqlserver_buffer_manager_page_life_expectancy
    - object: "SQLServer:Memory Manager"
      counters:
        - name: "Memory Grants Pending"
          field: sqlserver_memory_manager_memory_grants_pending

This is an example of the "memory available" output from metricbeat:

{
  "@timestamp": "2024-03-28T18:36:15.384Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "8.13.0"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "host": {
    "name": "server",
    "hostname": "server",
    "architecture": "x86_64",
    "os": {
      "kernel": "10.0.14393.6795 (rs1_release.240226-1752)",
      "build": "14393.6796",
      "type": "windows",
      "platform": "windows",
      "version": "10.0",
      "family": "windows",
      "name": "Windows Server 2016 Standard"
    },
    "id": "016f9785-5491-4884-9e5b-6aa4d77249fe",
    "ip": [
        "10.XXX.XXX.XXX"
    ],
    "mac": [
      "00-00-00-00-00-00"
    ]
  },
  "agent": {
    "version": "8.13.0",
    "ephemeral_id": "8e312c94-4bd2-4fbf-8bc8-cbd009262e01",
    "id": "ed6468c8-1d3b-41ad-aaad-7daf19dc7a4a",
    "name": "server",
    "type": "metricbeat"
  },
  "service": {
    "type": "windows"
  },
  "windows": {
    "perfmon": {
      "object": "Memory",
      "instance": "Memory",
      "metrics": {
        "memory_available_mbytes": 2229
      }
    }
  },
  "error": {
    "message": "failed on query=\\\\server\\Memory\\Available MBytes: %!w(<nil>)"
  },
  "event": {
    "duration": 2802996600,
    "dataset": "windows.perfmon",
    "module": "windows"
  },
  "metricset": {
    "period": 15000,
    "name": "perfmon"
  }
}