I am trying to use the logstash http_poller input plugin to ingest Azure Signin Logs (without an eventhub), then pass each document to logs-azure.signinlogs-1.22.0 ingest pipeline for proper parsing.
My current logstash config looks like this:
input {
http_poller {
urls => {
test1 => "https://endpoint"
}
request_timeout => 250
schedule => { "every"=> "5m"}
codec => json
}
}
filter {}
output {
elasticsearch {
cloud_id => "id"
cloud_auth => "pass"
data_stream => "true"
data_stream_type => "a"
data_stream_dataset => "b"
data_stream_namespace => "c"
validate_after_inactivity => 0
pipeline => "logs-azure.signinlogs-1.22.0"
}
}
The data coming from the endpoint looks like this:
[
{
"id": "uuid",
"createdDateTime": "2023-01-13T10:4:27Z",
"userDisplayName": "John Doe",
"userPrincipalName": "jdoe@me.com",
"userId": "uuid",
"appId": "uuid",
"riskEventTypes": [],
"riskEventTypes_v2": [],
"resourceDisplayName": "AppName",
"resourceId": "uuid",
"status": {
"errorCode": 0,
"failureReason": "Other.",
"additionalDetails": "MFA"
},
"deviceDetail": {
"deviceId": "",
"isManaged": false,
"trustType": ""
},
"appliedConditionalAccessPolicies": []
},
{
"id": "uuid",
"createdDateTime": "2023-10-10T11:33:12Z",
"userDisplayName": "John Doe",
"userPrincipalName": "jdoe@me.com",
"userId": "uuid",
"appId": "uuid",
"riskEventTypes": [],
"riskEventTypes_v2": [],
"resourceDisplayName": "AppName",
"resourceId": "uuid",
"status": {
"errorCode": 0,
"failureReason": "Other.",
"additionalDetails": "MFA"
},
"deviceDetail": {
"deviceId": "",
"isManaged": false,
"trustType": ""
},
"appliedConditionalAccessPolicies": []
}
]
When I run the pipeline, I get this error:
field [message] not present as part of path [message]
Then I updated the http_poller input plugin to include:
codec => json { target => "message" }
ecs_compatibility => v1
Which essentially encapsulates the entire document at the root level, when I rerun logstash, now I get errors that look like this:
Unexpected character ('r' (code 114)): was expecting double-quote to start field name\n at [Source: (String)"{resourceDisplayName=Value, resourceId=uuid, appliedConditionalAccessPolicies=, riskEventTypes=, deviceDetail={displayName=, browser=Value 7.32.91, trustType=, deviceId=, operatingSystem=Value, isCompliant=false, isManaged=false}, appDisplayName=Azure Virtual Desktop Client, isInteractive=true, riskLevelDuringSignIn=none, conditionalAccessStatus=success, ipAddress=IPv6Addr, createdDateTime=2025-03-11T03:10:31Z, u"[truncated 714 chars]; line: 1, column: 2]
I'm not entirely sure what the issue is since what I got from the endpoint is a valid JSON.
Any suggestion is appreciated.
Thanks,
O_O_O
