Azure storage metricset - elasticsearch receiving partial data

Hi i'm using metricbeat 7.6.2. I'm using Azure metricset to monitor all storage namespaces [blob, file ,queue, table]. This is my metricbeat Azure configuration:

metricbeat.modules:
- module: azure
  metricsets:
  - storage
  enabled: true
  period: 300s
  client_id: '${AZURE_CLIENT_ID:""}'
  client_secret: '${AZURE_CLIENT_SECRET:""}'
  tenant_id: '${AZURE_TENANT_ID:""}'
  subscription_id: '${AZURE_SUBSCRIPTION_ID:""}'
  refresh_list_interval: 600s

When running metricbeat on debug mode i can see that all namespaces are being sent but for some reason only "Microsoft.Storage/storageAccounts" and "Microsoft.Storage/storageAccounts/blobServices" and limited metrics [availability, egress, ingress, successe2elatency, successserverlatency, transactions] are arriving to my elasticsearch instance. anyone got an idea what can cause this issue?

Some examples of metricbeat debug output that does not arrive to my elasticsearch:

2020-05-07T11:59:26.839+0300    DEBUG   [processors]    processing/processors.go:186    Publish event: {
  "@timestamp": "2020-05-07T06:59:00.000Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.6.2"
  },
  "metricset": {
    "name": "storage",
    "period": 300000
  },
  "service": {
    "type": "azure"
  },
  "ecs": {
    "version": "1.4.0"
  },
  "event": {
    "dataset": "azure.storage",
    "module": "azure",
    "duration": 91433694547
  },
  "azure": {
    "resource": {
      "type": "Microsoft.Storage/storageAccounts",
      "group": "roni-new-branch",
      "name": "logziologsads6y3ltgo3y4"
    },
    "subscription_id": "72170a3a-d85e-404e-88df-4c4f33648561",
    "namespace": "Microsoft.Storage/storageAccounts/fileServices",
    "storage": {
      "filecapacity": {
        "avg": 0
      },
      "filecount": {
        "avg": 0
      },
      "filesharecount": {
        "avg": 0
      },
      "filesharesnapshotcount": {
        "avg": 0
      },
      "filesharesnapshotsize": {
        "avg": 0
      },
      "filesharecapacityquota": {
        "avg": 0
      }
    }
  },
  "cloud": {
    "region": "westus",
    "provider": "azure"
  },
  "logzio_codec": "json",
  "token": "",
  "host": {
    "name": "ip-192-168-0-6.ec2.internal"
  },
  "agent": {
    "ephemeral_id": "da669fbf-eb2e-4f37-8a17-a79a4e3068ec",
    "hostname": "ip-192-168-0-6.ec2.internal",
    "id": "636f1303-8f52-445e-8fce-d0a2f7c6907a",
    "version": "7.6.2",
    "type": "metricbeat"
  }
}

2020-05-07T11:59:26.840+0300    DEBUG   [processors]    processing/processors.go:186    Publish event: {
  "@timestamp": "2020-05-07T06:59:00.000Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.6.2"
  },
  "host": {
    "name": "ip-192-168-0-6.ec2.internal"
  },
  "agent": {
    "id": "636f1303-8f52-445e-8fce-d0a2f7c6907a",
    "version": "7.6.2",
    "type": "metricbeat",
    "ephemeral_id": "da669fbf-eb2e-4f37-8a17-a79a4e3068ec",
    "hostname": "ip-192-168-0-6.ec2.internal"
  },
  "cloud": {
    "provider": "azure",
    "region": "westus"
  },
  "metricset": {
    "period": 300000,
    "name": "storage"
  },
  "ecs": {
    "version": "1.4.0"
  },
  "token": "",
  "logzio_codec": "json",
  "azure": {
    "resource": {
      "name": "logziologsads6y3ltgo3y4",
      "type": "Microsoft.Storage/storageAccounts",
      "group": "roni-new-branch"
    },
    "subscription_id": "72170a3a-d85e-404e-88df-4c4f33648561",
    "namespace": "Microsoft.Storage/storageAccounts/blobServices",
    "storage": {
      "blobcount": {
        "avg": 0
      },
      "containercount": {
        "avg": 0
      },
      "indexcapacity": {
        "avg": 0
      },
      "blobcapacity": {
        "avg": 0
      }
    }
  },
  "service": {
    "type": "azure"
  },
  "event": {
    "duration": 91434378996,
    "dataset": "azure.storage",
    "module": "azure"
  }
}


2020-05-07T11:59:26.840+0300    DEBUG   [processors]    processing/processors.go:186    Publish event: {
  "@timestamp": "2020-05-07T06:59:00.000Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.6.2"
  },
  "metricset": {
    "name": "storage",
    "period": 300000
  },
  "host": {
    "name": "ip-192-168-0-6.ec2.internal"
  },
  "cloud": {
    "provider": "azure",
    "region": "westus"
  },
  "azure": {
    "resource": {
      "type": "Microsoft.Storage/storageAccounts",
      "group": "roni-new-branch",
      "name": "logziologsads6y3ltgo3y4"
    },
    "subscription_id": "72170a3a-d85e-404e-88df-4c4f33648561",
    "namespace": "Microsoft.Storage/storageAccounts/tableServices",
    "storage": {
      "tableentitycount": {
        "avg": 1058
      },
      "tablecapacity": {
        "avg": 3.08162e+06
      },
      "tablecount": {
        "avg": 5
      }
    }
  },
  "service": {
    "type": "azure"
  },
  "token": "",
  "logzio_codec": "json",
  "agent": {
    "version": "7.6.2",
    "type": "metricbeat",
    "ephemeral_id": "da669fbf-eb2e-4f37-8a17-a79a4e3068ec",
    "hostname": "ip-192-168-0-6.ec2.internal",
    "id": "636f1303-8f52-445e-8fce-d0a2f7c6907a"
  },
  "ecs": {
    "version": "1.4.0"
  },
  "event": {
    "duration": 91434710168,
    "dataset": "azure.storage",
    "module": "azure"
  }
}

hi @Yotamloe, from the DEBUG messages we see these events are published, can you check in the elasticsearch logs for any errors there, on why the events are not being ingested?