Trying to understand where is the issue and what can be done regarding parsing and information further usage. Azure Activity Logs are streamed to Event Hub from where Standalone Agent (version 8.12.1) picks them up and ingests into Elasticsearch.
Configured integration:
inputs:
- id: azure-eventhub-activitylogs
type: azure-eventhub
use_output: default
data_stream:
namespace: default
streams:
- data_stream:
dataset: azure.activitylogs
type: logs
elasticsearch:
dynamic_dataset: true
connection_string: "xxx"
storage_account_container: "xxx"
eventhub: "xxx"
consumer_group: "xxx"
storage_account: "xxx"
storage_account_key: "xxx"
tags:
- azure-activitylogs
Trying to understand what can be done regarding activity events message. Currently created document is not even similar to the one provided in example: Azure Activity Logs | Documentation
Created document structure using this integration:
{
"_index": "xxx",
"_id": "xxx",
"_version": 1,
"_score": 0,
"_source": {
"@timestamp": "xxx",
"event": {
"dataset": "azure.activitylogs"
},
"elastic_agent": {
"version": "8.12.1",
"id": "xxx",
"snapshot": false
},
"ecs": {
"version": "8.0.0"
},
"cloud": {
"region": "westeurope",
"provider": "azure",
"account": {
"id": "xxx"
},
"instance": {
"name": "xxx",
"id": "xxx"
},
"machine": {
"type": "xxx"
},
"service": {
"name": "xxx"
}
},
"message": "{\"ReleaseVersion\":\"xx.xx.xx\",\"RoleLocation\":\"West Europe\",\"callerIpAddress\":\"xx.xx.xxx.xxx\",\"category\":\"Administrative\",\"correlationId\":\"xxx\",\"durationMs\":\"0\",\"identity\":{\"authorization\":{\"action\":\"Microsoft.ManagedIdentity/userAssignedIdentity/write\",...}}}",
"azure": {
"enqued_time": "xxx",
"eventhub": "xxx",
"consumer_group": "$Default",
"offset": xxxx,
"sequence_number": xxx
},
"tags": [
"azure-activitylogs",
],
"data_stream": {
"namespace": "default",
"type": logs,
"dataset": "azure.activitylogs"
},
"agent": {
"version": "8.12.1",
"ephemeral_id": "xxx",
"id": "xxx",
"name": "xxx",
"type": "filebeat",
},
"Similarly, here follows some JSON format information related to cloud, host.."
}
}
However, something does not look well to me. What are options to parse "message" and use it's content for further usage in dashboards? The aim of this integration usage is to obtain events from Identity Management, Service Principal (configured diagnostic settings to stream Azure Activity Logs to event hub, then Agent picks them up). Was expecting document content similar to provided example in Azure Activity Logs integration documentation. Did I miss something crucial in configuration?
