Hi,
we are running Elastic Cloud and want to add Application Logs to it with Elastic Agent, these logs need to be correlated with out APM traces.
We are running the Elastic Azure Logs Integration with the Elastic Agent running on a vm.
Our applications log in the ECS format and the data is sent do an Event Hub and also shows up in Elastic and is correlated accordingly.
However, the message in Elastic consists of the whole event-hub JSON and fields like service.name and log.level are empty.
Am I missing something on how this can be added the correct way to elastic (message -> message filed ecc.) Or is this not possible at the moment?
I did try around with some custom pipelines, but this seems cumbersome.
The message in elastic:
Note that resultDescription is our ECS logged message
{
"EventIpAddress": "10.81.0.18",
"EventPrimaryStampName": "waws-prod-fra-013",
"EventStampName": "waws-prod-fra-013",
"EventStampType": "Stamp",
"Host": "zzz",
"category": "AppServiceConsoleLogs",
"containerId": "yyy",
"level": "Informational",
"location": "Germany West Central",
"operationName": "Microsoft.Web/sites/log",
"resourceId": "/SUBSCRIPTIONS/xxx",
"resultDescription": " {\"@timestamp\":\"2023-04-28T09:15:17.317Z\",\"log.level\":\"warn\",\"message\":\"ApplicationError: [409] Entity exist already\",\"ecs\":{\"version\":\"1.6.0\"},\"event\":{\"dataset\":\"service-name.log\"},\"trace\":{\"id\":\"702702a79eaadb6ec76bc5b9c84a288a\"},\"transaction\":{\"id\":\"eac23a691b53e73d\"},\"service\":{\"name\":\"service-name\"}}\n\n",
"time": "2023-04-28T09:15:17.317964487Z"
}
An example log generated by our app in ECS format:
{
"@timestamp": "2023-04-28T09:08:33.060Z",
"log.level": "warn",
"message": "ApplicationError: [409] Entity exist already",
"ecs": {
"version": "1.6.0"
},
"event": {
"dataset": "service-name.log"
},
"trace": {
"id": "17eeb484574b0075a9607c69f07c42bd"
},
"transaction": {
"id": "b38a1d4d39f1f8ba"
},
"service": {
"name": "service-name"
}
}