Issue parsing azure platform properties

Elastic Stack Elastic Agent
1

I am also working with support on this but was hoping to get a faster response here. Anyone using Azure Eventhubs forAppServiceHTTPLogs, I can get the logs to come in but I need to parse the azure.platform.properties field and I have a custom pipeline with the first processor stating that the field is JSON but when I review the logs in discovery I am getting this error.

Processor "json" with tag "" in pipeline "logs-azure.platformlogs@custom" failed with message "Unexpected character ('C' (code 67)): was expecting double-quote to start field name\n at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 3]"

Here is the field as it comes in

{
"Cookie": "-",
"CsBytes": 1249,
"SPort": "443",
"TimeTaken": 15,
"Referer": "-",
"ScWin32Status": "0",
"CIp": "1.1.1.18",
"Result": "Success",
"CsHost": "REDACTED",
"ScBytes": 606,
"CsUriQuery": "id=2&returnToUrl=/Contacts/Contacts&X-ARR-LOG-ID=73f9ec26-b6c5-400f-9e56-d97bac83aca2",
"CsMethod": "GET",
"ComputerName": "REDACTED",
"UserAgent": "Mozilla/5.0+AppleWebKit/537.36+(KHTML,+like+Gecko;+compatible;+bingbot/2.0;++http://www.bing.com/bingbot.htm)+Chrome/103.0.5060.134+Safari/537.36",
"ScSubStatus": "0",
"CsUriStem": "/Contacts/Contact/Details",
"CsUsername": "-",
"ScStatus": 302
}
1 Like
2

Hi @intsec
Is the field really "pretty" JSON (multiple lines) most of the parsers are expecting compact / ndjson perhaps that is the issue.

Can you see the raw logs? Can you configure the logs to be compact / ndjson?

Just something to check

3

I have the JSON listed, it looks to be JSON when I look at it.

4

Yes it is JSON

a) This is pretty JSON (i.e. multiple lines)

{
  "foo" : "bar"
}

b) This is compact / ndjson single line per document,

{ "foo" : "bar" }

Most the streaming parsers are looking for type b that is what I am asking...

5

Ahh ok, I understand you now and I have seen this being talked about and people were saying you need to somehow change it to ndjson before it hits elasticsearch? If so I am not sure how to do this.

6

I would check with the Azure forum, docs etc to see if you can emit in compact form

Or in our integration is there a flag, which exact integration are you using...

7

Azure Logs under the Platform Logs/

8

can you be more specific ..

We have Filebeat and Agent, Which exact Integration,Which Exact Version, Screenshot etc. ... We can help more when you are very specific...

We answer 100s of questions...help us .. help you...

Sharing Configurations (sanitized is also helpful)

9

Certainly,
We are using only agent, and its the Azure Logs integration

elastic-screenshot1
elastic-screenshot11090×211 23.8 KB

and the HTTPLogs come in through the Platform Logs section under the Azure Logs integration configuration

elastic-screenshot2
elastic-screenshot2818×657 55.9 KB

elastic-screenshot3
elastic-screenshot3831×618 71.7 KB

We are using the integration version 1.5.13 for Azure Logs. Elastic v 8.5.3

10

Ok cool thanks

Interestingly the HTTP logs you are showing do not look like the examples in the docs here

The log you are showing looks to be a log from an HTTP Server (as you note)

The easiest thing would probably be to track that down and have them adjust their output to ndjson (which is very common these days)

Not to say you can not fix this but you are going to put in a multi-line filter etc, which is more complex .. and might interfere with other types of logs...

Perhaps a friend of mine might have an idea

@hemantmalik Any Thoughts

11

When I started the ticket with Elastic on this they had said they were looking into adding support for the HTTPLogs that come from Azure HTTP service which I have a client that uses for their web application. So that could be why you don't see examples specifically for the HTTP Logs.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.