How to read log json array format with filebeat

Elastic Stack Beats
1

I have many log look like (json array)
This is raw log

[{"CreationTime":"2017-06-17T06:28:56","Id":"77bf08c6-5bb9-4333-a394-03f181d7250a","Operation":"UserLoggedIn","OrganizationId":"4ebc33361-871a-44c5-93a5-60eb590917cd","RecordType":15,"ResultStatus":"Succeeded","UserKey":"10037FFE9D6C9759@fpt.com.vn","UserType":0,"Version":1,"Workload":"AzureActiveDirectory","ClientIP":"116.96.81.65","ObjectId":"00000002-0000-0ff1-ce00-000000000000","UserId":"ManhNT16@company.com","AzureActiveDirectoryEventType":1,"ExtendedProperties":[{"Name":"UserAgent","Value":"Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"},{"Name":"UserAuthenticationMethod","Value":"65"},{"Name":"RequestType","Value":"WsFederation:wsfederation"},{"Name":"ResultStatusDetail","Value":"Success"}],"Actor":[{"ID":"f6a73d23-ddba-4333-9106-08efaee81e0e","Type":0},{"ID":"ManhNT16@company.com","Type":5},{"ID":"10037FFE9D6C9759","Type":3}],"ActorContextId":"4ebc9261-333a-44c5-93a5-60eb590917cd","ActorIpAddress":"116.96.81.65","InterSystemsId":"2948868c-6f8a-4b13-a44a-8011b254559d","IntraSystemId":"d5fef7d3-3337-45a2-b91b-b8f004ae1300","Target":[{"ID":"00000002-0000-0ff1-ce00-000000000000","Type":0}],"TargetContextId":"4ebc9261-871a-44c5-93a5-60eb590917cd","ApplicationId":"00000002-0000-0ff1-ce00-000000000000"}]

How to make filebeat read log above ? I know that filebeat does not start with [ and combines them with the previous line that does.
I add some config but it's not work. Filebeat can't read log file

multiline.pattern: '^['
multiline.negate: true
multiline.match: after

And here is friendly log

[
	{
		"CreationTime": "2017-06-17T06:28:56",
		"Id": "77bf08c6-5bb9-41d1-a394-03f181d7250a",
		"Operation": "UserLoggedIn",
		"OrganizationId": "4ebc9261-871a-44c5-93a5-60eb333317cd",
		"RecordType": 15,
		"ResultStatus": "Succeeded",
		"UserKey": "10037AAA9D6C9759@mycompany.com",
		"UserType": 0,
		"Version": 1,
		"Workload": "AzureActiveDirectory",
		"ClientIP": "116.96.81.65",
		"ObjectId": "00000002-1111-0ff1-ce00-000000000000",
		"UserId": "ManhNT16@mycompany.com",
		"AzureActiveDirectoryEventType": 1,
		"ExtendedProperties": [
			{
				"Name": "UserAgent",
				"Value": "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
			},
			{
				"Name": "UserAuthenticationMethod",
				"Value": "65"
			},
			{
				"Name": "RequestType",
				"Value": "WsFederation:wsfederation"
			},
			{
				"Name": "ResultStatusDetail",
				"Value": "Success"
			}
		],
		"Actor": [
			{
				"ID": "f6a73d23-ddba-4552-9106-08efeee81e0e",
				"Type": 0
			},
			{
				"ID": "ManhNT16@mycompany.com",
				"Type": 5
			},
			{
				"ID": "10037AAA9D6C9759",
				"Type": 3
			}
		],
		"ActorContextId": "4ebc9261-871a-44c5-93a5-60eb333917cd",
		"ActorIpAddress": "116.96.81.65",
		"InterSystemsId": "2948868c-6f8a-4b13-a44a-8011b334559d",
		"IntraSystemId": "d5fef7d3-4417-45a2-b91b-b8f033ae1300",
		"Target": [
			{
				"ID": "00000002-0000-0ff1-ce00-000000000000",
				"Type": 0
			}
		],
		"TargetContextId": "4ebc9261-871a-44c5-93a5-60eb333917cd",
		"ApplicationId": "00000002-0000-0ff1-ce00-000000000000"
	}
]
2

Anyone help me? ... Thanks!

3

Can you share your full filebeat config? Make sure the multiline section is correctly put with the prospector.

4

Thanks @steffens, this is my filebeat config

filebeat.prospectors:
- input_type: log
  paths:
    - G:\log\Audit.AzureActiveDirectory
  document_type: azure
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after

- input_type: log
  paths:
    - G:\log\Audit.Exchange
  document_type: exchange
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after

- input_type: log
  paths:
    - G:\log\Audit.General
  document_type: general
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after

- input_type: log
  paths:
    - G:\log\Audit.SharePoint
  document_type: sharepoint
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["10.1.6.22:5044", "10.1.6.23:5044"]
  worker: 2
  compression_level: 3
  loadbalance: true
  index: 'fb-fim-office'
5

I guess your log format looks like in this tester: https://play.golang.org/p/cP2qjYwYf7

The patterns looks ok. Are there multiple events in a log file? With multiple events, are they put into separate arrays, or all in the top-level array? Is the log pretty printed or not?

6

Yes. have there are multiple event in log file

It is separate arrays.

No. It look like raw look above.

Thanks you!

7

Do you have a sample log (multiple events)? I have a hard time tell how logs look like and how processing could be done.

8

Sorry, i checked my log, it have only one array in one file and have many file :slight_smile:

All in the top-level array. Like this. It have multiple events.

https://pastebin.com/raw/AXh80s3m

9

@steffens Can u help me solve my problem? Thanks !

10

I checked a hexdump of your logfile. The problem is not multiline. Actually you don't need multiline for this log file. I see some interesting things in your log file:

  1. No newline character at the end. The newline is required by filebeat to determine a log line is complete.
  2. Some wird binary sequence (non-printable characters?) At the end of your file (last 10 bytes)
  3. Not sure if I'm missing some fonts, but I can not correctly see some of the embedded strings with default UTF-8 encoding in my editor. Seems like windows path being included as is, in UTF-16 encoding. E.g. is this path correct? \\MuÌ£c Ä‘ã xoÌ (looks like some generated path to store some actual mail content in)?

I guess these files are written at once and never been touched afterwards?

11

Thank @steffens. I sent to you my answer via inbox messagage.

Not sure if I’m missing some fonts, but I can not correctly see some of the embedded strings with default UTF-8 encoding in my editor. Seems like windows path being included as is, in UTF-16 encoding. E.g. is this path correct? \MuÌ£c Ä‘ã xoÌ (looks like some generated path to store some actual mail content in)?

This path is correctly. It generate by O365. I dont know that why. Maybe Log )365 dont support my language.

I guess these files are written at once and never been touched afterwards?

Yeah, i dont touch anything.