Azure Function's log streams from Azure Event Hub can't be consumed by Filebeat

Elastic Stack Beats
1

I enable azure module to streams azure platformlog to Elasticsearch.

When I collected the logs of azure postgreSQL and aks, everything worked fine, but when I used the same method to collect the logs of azure function, something went wrong .

I found that the log format of the azure function is not the same as that of the other two services

As shown in the following code example, we can find that in the properties attribute, it is an illegal json string with single quotes

 {
    "records": [
      {
        "level": "Informational",
        "resourceId": "/SUBSCRIPTIONS/<my subscription id>/RESOURCEGROUPS/<my resource group>/PROVIDERS/MICROSOFT.WEB/SITES/<my azure function>"
        "operationName": "Microsoft.Web/sites/functions/log",
        "category": "FunctionAppLogs",
        "time": "01/04/2023 01:55:00",
        "properties": "{'appName':'<my azure function>','roleInstance':'<id>','message':'Host Status: {\\n  \\'id\\': \\'<function app id>\\',\\n  \\'state\\': \\'Running\\',\\n  \\'version\\': \\'4.13.0.0\\',\\n  \\'versionDetails\\': \\'4.13.0+da9a765ed67be48c79440526f78fa1b5c6efdeea\\',\\n  \\'platformVersion\\': \\'99.0.10.764\\',\\n  \\'instanceId\\': \\'<instance id>\\',\\n  \\'computerName\\': \\'<computer name>\\',\\n  \\'processUptime\\': 69254486,\\n  \\'functionAppContentEditingState\\': \\'Unknown\\'\\n}','category':'Host.Controllers.Host','hostVersion':'4.13.0.0','hostInstanceId':'<host id>','level':'Information','levelId':2,'processId':1}",
        "EventStampType": "Stamp",
        "EventPrimaryStampName": "waws-prod-ty1-081",
        "EventStampName": "waws-prod-ty1-081",
        "Host": "<host name>",
        "EventIpAddress": "<ip address>"
      }
}

Below is the log of the service that can normally collect logs through filebeat

{
    "records": [
      {
        "time": "2023-01-04T03:45:31.1040000Z",
        "properties": {
          "timestamp": "2023-01-04 03:45:31.104 UTC",
          "processId": 8909,
          "errorLevel": "LOG",
          "sqlerrcode": "00000",
          "message": "2023-01-04 03:45:31 UTC-63b4f65b.22cd-LOG:  connection received: host=<host> port=<port>"
        },
        "resourceId": "/SUBSCRIPTIONS/<my subscription id>/RESOURCEGROUPS/<my resource group>/PROVIDERS/MICROSOFT.DBFORPOSTGRESQL/FLEXIBLESERVERS/<postgres server>",
        "category": "PostgreSQLLogs",
        "operationName": "LogEvent"
      }
    ]
  }

We can find that the properties attribute in the above code example is the expected flattened json.

I would like to ask if there are two different formats in the same source and one of them is an illegal json string. Is there a way to deal with it by configuring the filebeat processor?

Does anyone have relevant experience to share? Thanks .

reference:
Invalid JSON logs produced for function apps. - Microsoft Q&A

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.