Azure storage metricset - elasticsearch receiving partial data

Hi i'm using metricbeat 7.6.2. I'm using Azure metricset to monitor all storage namespaces [blob, file ,queue, table]. This is my metricbeat Azure configuration:

metricbeat.modules:
- module: azure
  metricsets:
  - storage
  enabled: true
  period: 300s
  client_id: '${AZURE_CLIENT_ID:""}'
  client_secret: '${AZURE_CLIENT_SECRET:""}'
  tenant_id: '${AZURE_TENANT_ID:""}'
  subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'
  refresh_list_interval: 600s

When running metricbeat on debug mode i can see that all namespaces are being sent but for some reason only "Microsoft.Storage/storageAccounts" and "Microsoft.Storage/storageAccounts/blobServices" and limited metrics [availability, egress, ingress, successe2elatency, successserverlatency, transactions] are arriving to my elasticsearch instance. anyone got an idea what can cause this issue?

Some examples of metricbeat debug output that does not arrive to my elasticsearch:

2020-05-07T11:59:26.839+0300    DEBUG   [processors]    processing/processors.go:186    Publish event: {
  "@timestamp": "2020-05-07T06:59:00.000Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.6.2"
  },
  "metricset": {
    "name": "storage",
    "period": 300000
  },
  "service": {
    "type": "azure"
  },
  "ecs": {
    "version": "1.4.0"
  },
  "event": {
    "dataset": "azure.storage",
    "module": "azure",
    "duration": 91433694547
  },
  "azure": {
    "resource": {
      "type": "Microsoft.Storage/storageAccounts",
      "group": "roni-new-branch",
      "name": "logziologsads6y3ltgo3y4"
    },
    "subscription_id": "72170a3a-d85e-404e-88df-4c4f33648561",
    "namespace": "Microsoft.Storage/storageAccounts/fileServices",
    "storage": {
      "filecapacity": {
        "avg": 0
      },
      "filecount": {
        "avg": 0
      },
      "filesharecount": {
        "avg": 0
      },
      "filesharesnapshotcount": {
        "avg": 0
      },
      "filesharesnapshotsize": {
        "avg": 0
      },
      "filesharecapacityquota": {
        "avg": 0
      }
    }
  },
  "cloud": {
    "region": "westus",
    "provider": "azure"
  },
  "logzio_codec": "json",
  "token": "",
  "host": {
    "name": "ip-192-168-0-6.ec2.internal"
  },
  "agent": {
    "ephemeral_id": "da669fbf-eb2e-4f37-8a17-a79a4e3068ec",
    "hostname": "ip-192-168-0-6.ec2.internal",
    "id": "636f1303-8f52-445e-8fce-d0a2f7c6907a",
    "version": "7.6.2",
    "type": "metricbeat"
  }
}

2020-05-07T11:59:26.840+0300    DEBUG   [processors]    processing/processors.go:186    Publish event: {
  "@timestamp": "2020-05-07T06:59:00.000Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.6.2"
  },
  "host": {
    "name": "ip-192-168-0-6.ec2.internal"
  },
  "agent": {
    "id": "636f1303-8f52-445e-8fce-d0a2f7c6907a",
    "version": "7.6.2",
    "type": "metricbeat",
    "ephemeral_id": "da669fbf-eb2e-4f37-8a17-a79a4e3068ec",
    "hostname": "ip-192-168-0-6.ec2.internal"
  },
  "cloud": {
    "provider": "azure",
    "region": "westus"
  },
  "metricset": {
    "period": 300000,
    "name": "storage"
  },
  "ecs": {
    "version": "1.4.0"
  },
  "token": "",
  "logzio_codec": "json",
  "azure": {
    "resource": {
      "name": "logziologsads6y3ltgo3y4",
      "type": "Microsoft.Storage/storageAccounts",
      "group": "roni-new-branch"
    },
    "subscription_id": "72170a3a-d85e-404e-88df-4c4f33648561",
    "namespace": "Microsoft.Storage/storageAccounts/blobServices",
    "storage": {
      "blobcount": {
        "avg": 0
      },
      "containercount": {
        "avg": 0
      },
      "indexcapacity": {
        "avg": 0
      },
      "blobcapacity": {
        "avg": 0
      }
    }
  },
  "service": {
    "type": "azure"
  },
  "event": {
    "duration": 91434378996,
    "dataset": "azure.storage",
    "module": "azure"
  }
}


2020-05-07T11:59:26.840+0300    DEBUG   [processors]    processing/processors.go:186    Publish event: {
  "@timestamp": "2020-05-07T06:59:00.000Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.6.2"
  },
  "metricset": {
    "name": "storage",
    "period": 300000
  },
  "host": {
    "name": "ip-192-168-0-6.ec2.internal"
  },
  "cloud": {
    "provider": "azure",
    "region": "westus"
  },
  "azure": {
    "resource": {
      "type": "Microsoft.Storage/storageAccounts",
      "group": "roni-new-branch",
      "name": "logziologsads6y3ltgo3y4"
    },
    "subscription_id": "72170a3a-d85e-404e-88df-4c4f33648561",
    "namespace": "Microsoft.Storage/storageAccounts/tableServices",
    "storage": {
      "tableentitycount": {
        "avg": 1058
      },
      "tablecapacity": {
        "avg": 3.08162e+06
      },
      "tablecount": {
        "avg": 5
      }
    }
  },
  "service": {
    "type": "azure"
  },
  "token": "",
  "logzio_codec": "json",
  "agent": {
    "version": "7.6.2",
    "type": "metricbeat",
    "ephemeral_id": "da669fbf-eb2e-4f37-8a17-a79a4e3068ec",
    "hostname": "ip-192-168-0-6.ec2.internal",
    "id": "636f1303-8f52-445e-8fce-d0a2f7c6907a"
  },
  "ecs": {
    "version": "1.4.0"
  },
  "event": {
    "duration": 91434710168,
    "dataset": "azure.storage",
    "module": "azure"
  }
}

hi @Yotamloe, from the DEBUG messages we see these events are published, can you check in the elasticsearch logs for any errors there, on why the events are not being ingested?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.