Hi everyone,
I'm trying the filebeat beta feature to pull azure platform logs into ES and I'm hitting ing the following issue:
When there is an error message on the APIM logs, the field lastError is being added to the azure log properties as such:
{
Fields: {
"agent": {
"type": "filebeat",
"version": "7.11.1"
},
"ecs": {
"version": "1.6.0"
},
"event": {
"dataset": "azure.platformlogs",
"module": "azure"
},
"fileset": {
"name": "platformlogs"
},
"input": {
"type": "azure-eventhub"
},
"message": "{\"Level\":4,\"callerIpAddress\":\"ipHere\",\"category\":\"GatewayLogs\",\"correlationId\":\"idHere\",\"durationMs\":0,\"isRequestSuccess\":false,\"location\":\"North Central US\",\"operationName\":\"Microsoft.ApiManagement/GatewayLogs\",\"properties\":{\"apiId\":\"api-id\",\"apiRevision\":\"1\",\"cache\":\"none\",\"clientProtocol\":\"HTTP/1.1\",\"clientTlsVersion\":\"1.2\",\"lastError\":{\"message\":\"Claim value mismatch: email=sampleemail@something.com.\",\"path\":\"choose\\\\when[1]\",\"reason\":\"TokenClaimValueMismatch\",\"scope\":\"api\",\"section\":\"inbound\",\"source\":\"validate-jwt\"},\"method\":\"GET\",\"operationId\":\"GET\",\"responseCode\":401,\"responseSize\":316,\"url\":\"https://some.url.here\"},\"resourceId\":\"/resourceIdHere\",\"time\":\"2021-03-02T10:15:00.2443029Z\"}",
"service": {
"type": "azure"
}
}
}
However, when I search through my logs using Kibana I cannot find the lastError field and the data included inside it. Basically, that information is lost. All the other message fields, including the other property fields are being correctly shown, only the fields of the lastError json are being lost. The following message appears on the filebeat logs.
2021-03-02T10:16:45.987Z WARN [elasticsearch] elasticsearch/client.go:408 Cannot index event publisher.Event {azure log content here with lastError field included}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [azure.platformlogs.properties.lastError] of type [keyword] in document with id 'qLxw8ncBPAyIKXCBqjeG'. Preview of field's value: '{path=choose\\when[1], reason=TokenClaimValueMismatch, scope=api, section=inbound, source=validate-jwt, message=Claim value mismatch: email=sampleemail@something.com.}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:1127"}}
PS: The lastError field is only present inside the properties when there is an error on the APIM side. When a successful request is being logged then this field will not be present inside the properties json.
Do you know if this could be a config issue on my side or a filebeat bug?