Hello,
I am new to elastic filebeat and need some assistance with an error. We wrote a powershell script to pull down Office 365 audit logs and write them out to CSV files. These records also include a field that contains all of the event details in json format, which I am writing out to a separate json file. I am trying to use filebeat to ingest those json files to our SIEM server (Humio) but am getting the error below in /logs/filebeat.
Any help is appreciated.
Thanks.
/logs/filebeat:
|2019-11-26T12:05:49.956-0500|INFO|log/harvester.go:251|Harvester started for file: D:\scripts\Office365AuditLogs\json\AuditRecords-20191126-1220.json|
|---|---|---|---|
|2019-11-26T12:05:49.956-0500|ERROR|readjson/json.go:52|Error decoding JSON: invalid character '\x00' looking for beginning of value|
|2019-11-26T12:05:49.956-0500|ERROR|readjson/json.go:52|Error decoding JSON: invalid character '\x00' looking for beginning of value|
This is the contents of the AuditRecords-20191126-1220.json file (this particular file only has only two lines but the above error is happening for all the files):
{"CreationTime":"2019-11-26T12:02:14","Id":"a9575928-0591-4336-eb55-08d7726879d6","Operation":"FileAccessed","OrganizationId":"[org_id]","RecordType":6,"UserKey":"[userkey]@live.com","UserType":0,"Version":1,"Workload":"OneDrive","ClientIP":"1.1.1.100","ObjectId":"https:my.sharepoint.com\/[url].docx","UserId":"user@email.com","CorrelationId":"correlation_id","EventSource":"SharePoint","ItemType":"File","ListId":"[list_id]","ListItemUniqueId":"[listitemuniqueid]","Site":"[site]","UserAgent":"Microsoft Office Word\/16.0.11328.20468 (Windows\/10.0; Desktop WOW64; en-US; Desktop app;)","WebId":"[web_id]","SourceFileExtension":"docx","SiteUrl":"https:\/\/my.sharepoint.com\/url\/","SourceFileName":"[filename].docx","SourceRelativeUrl":"[url]"}
{"CreationTime":"2019-11-26T12:02:48","Id":"40871d2b-3077-483e-5b34-08d772688e78","Operation":"SearchQueryPerformed","OrganizationId":"[org_id]","RecordType":4,"UserKey":"[userkey]@live.com","UserType":0,"Version":1,"Workload":"SharePoint","ClientIP":"3.3.3.103","ObjectId":"object_id","UserId":"#user@email.com","CorrelationId":"correlation_id","EventSource":"SharePoint","ItemType":"Web","UserAgent":"Mozilla\/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident\/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0E; .NET4.0C; InfoPath.3)","EventData":"<ClientType>OneDriveWeb<\/ClientType>"}
And my filebeat.yml:
#======================== Filebeat inputs =============================
filebeat.inputs:
- type: log
enabled: true
paths:
- D:\scripts\Office365AuditLogs\json\*.json
encoding: plain
json.keys_under_root: true
fields:
app_id: office365_audit
queue.mem:
events: 8000
flush.min_events: 1000
flush.timeout: 1s
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["https://[fqdn]:8443/internalapi"] # -> gets redirected to ["$BASEURL/api/v1/ingest/elastic-bulk"]
ssl.verification_mode: none
username: joe_user
password: [api key]
compression_level: 5
bulk_max_size: 200
worker: 1