# How to read log json array format with filebeat
**URL:** https://discuss.elastic.co/t/how-to-read-log-json-array-format-with-filebeat/96220
**Category:** Beats
**Tags:** filebeat
**Created:** [August 8, 2017, 8:20am UTC](https://discuss.elastic.co/t/how-to-read-log-json-array-format-with-filebeat/96220 "2017-08-08T08:20:10Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@tatdat](https://discuss.elastic.co/u/tatdat)
#### Post date: [August 8, 2017, 8:20am UTC](https://discuss.elastic.co/t/how-to-read-log-json-array-format-with-filebeat/96220/1 "2017-08-08T08:20:10Z")
I have many log look like (json array)
This is raw log
> [{"CreationTime":"2017-06-17T06:28:56","Id":"77bf08c6-5bb9-4333-a394-03f181d7250a","Operation":"UserLoggedIn","OrganizationId":"4ebc33361-871a-44c5-93a5-60eb590917cd","RecordType":15,"ResultStatus":"Succeeded","UserKey":"10037FFE9D6C9759@fpt.com.vn","UserType":0,"Version":1,"Workload":"AzureActiveDirectory","ClientIP":"116.96.81.65","ObjectId":"00000002-0000-0ff1-ce00-000000000000","UserId":"ManhNT16@company.com","AzureActiveDirectoryEventType":1,"ExtendedProperties":[{"Name":"UserAgent","Value":"Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"},{"Name":"UserAuthenticationMethod","Value":"65"},{"Name":"RequestType","Value":"WsFederation:wsfederation"},{"Name":"ResultStatusDetail","Value":"Success"}],"Actor":[{"ID":"f6a73d23-ddba-4333-9106-08efaee81e0e","Type":0},{"[ID":"ManhNT16@company.com](mailto:ID%22:%22ManhNT16@company.com)","Type":5},{"ID":"10037FFE9D6C9759","Type":3}],"ActorContextId":"4ebc9261-333a-44c5-93a5-60eb590917cd","ActorIpAddress":"116.96.81.65","InterSystemsId":"2948868c-6f8a-4b13-a44a-8011b254559d","IntraSystemId":"d5fef7d3-3337-45a2-b91b-b8f004ae1300","Target":[{"ID":"00000002-0000-0ff1-ce00-000000000000","Type":0}],"TargetContextId":"4ebc9261-871a-44c5-93a5-60eb590917cd","ApplicationId":"00000002-0000-0ff1-ce00-000000000000"}]
How to make filebeat read log above ? I know that filebeat does not start with [ and combines them with the previous line that does.
I add some config but it's not work. Filebeat can't read log file
> multiline.pattern: '^['
> multiline.negate: true
> multiline.match: after
And here is friendly log
```
[
{
"CreationTime": "2017-06-17T06:28:56",
"Id": "77bf08c6-5bb9-41d1-a394-03f181d7250a",
"Operation": "UserLoggedIn",
"OrganizationId": "4ebc9261-871a-44c5-93a5-60eb333317cd",
"RecordType": 15,
"ResultStatus": "Succeeded",
"UserKey": "10037AAA9D6C9759@mycompany.com",
"UserType": 0,
"Version": 1,
"Workload": "AzureActiveDirectory",
"ClientIP": "116.96.81.65",
"ObjectId": "00000002-1111-0ff1-ce00-000000000000",
"UserId": "ManhNT16@mycompany.com",
"AzureActiveDirectoryEventType": 1,
"ExtendedProperties": [
{
"Name": "UserAgent",
"Value": "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
},
{
"Name": "UserAuthenticationMethod",
"Value": "65"
},
{
"Name": "RequestType",
"Value": "WsFederation:wsfederation"
},
{
"Name": "ResultStatusDetail",
"Value": "Success"
}
],
"Actor": [
{
"ID": "f6a73d23-ddba-4552-9106-08efeee81e0e",
"Type": 0
},
{
"ID": "ManhNT16@mycompany.com",
"Type": 5
},
{
"ID": "10037AAA9D6C9759",
"Type": 3
}
],
"ActorContextId": "4ebc9261-871a-44c5-93a5-60eb333917cd",
"ActorIpAddress": "116.96.81.65",
"InterSystemsId": "2948868c-6f8a-4b13-a44a-8011b334559d",
"IntraSystemId": "d5fef7d3-4417-45a2-b91b-b8f033ae1300",
"Target": [
{
"ID": "00000002-0000-0ff1-ce00-000000000000",
"Type": 0
}
],
"TargetContextId": "4ebc9261-871a-44c5-93a5-60eb333917cd",
"ApplicationId": "00000002-0000-0ff1-ce00-000000000000"
}
]
```
---
### Author:  [@tatdat](https://discuss.elastic.co/u/tatdat)
#### Post date: [August 10, 2017, 1:37am UTC](https://discuss.elastic.co/t/how-to-read-log-json-array-format-with-filebeat/96220/2 "2017-08-10T01:37:27Z")
Anyone help me? ... Thanks!
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [August 10, 2017, 1:19pm UTC](https://discuss.elastic.co/t/how-to-read-log-json-array-format-with-filebeat/96220/3 "2017-08-10T13:19:08Z")
Can you share your full filebeat config? Make sure the multiline section is correctly put with the prospector.
---
### Author:  [@tatdat](https://discuss.elastic.co/u/tatdat)
#### Post date: [August 11, 2017, 2:58am UTC](https://discuss.elastic.co/t/how-to-read-log-json-array-format-with-filebeat/96220/4 "2017-08-11T02:58:18Z")
Thanks @steffens, this is my filebeat config
```
filebeat.prospectors:
- input_type: log
paths:
- G:\log\Audit.AzureActiveDirectory
document_type: azure
multiline.pattern: '^\['
multiline.negate: true
multiline.match: after
- input_type: log
paths:
- G:\log\Audit.Exchange
document_type: exchange
multiline.pattern: '^\['
multiline.negate: true
multiline.match: after
- input_type: log
paths:
- G:\log\Audit.General
document_type: general
multiline.pattern: '^\['
multiline.negate: true
multiline.match: after
- input_type: log
paths:
- G:\log\Audit.SharePoint
document_type: sharepoint
multiline.pattern: '^\['
multiline.negate: true
multiline.match: after
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["10.1.6.22:5044", "10.1.6.23:5044"]
worker: 2
compression_level: 3
loadbalance: true
index: 'fb-fim-office'
```
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [August 11, 2017, 3:38pm UTC](https://discuss.elastic.co/t/how-to-read-log-json-array-format-with-filebeat/96220/5 "2017-08-11T15:38:18Z")
I guess your log format looks like in this tester: [https://play.golang.org/p/cP2qjYwYf7](https://play.golang.org/p/cP2qjYwYf7)
The patterns looks ok. Are there multiple events in a log file? With multiple events, are they put into separate arrays, or all in the top-level array? Is the log pretty printed or not?
---
### Author:  [@tatdat](https://discuss.elastic.co/u/tatdat)
#### Post date: [August 13, 2017, 1:35pm UTC](https://discuss.elastic.co/t/how-to-read-log-json-array-format-with-filebeat/96220/6 "2017-08-13T13:35:44Z")
> [@steffens](#):
>
> Are there multiple events in a log file?
Yes. have there are multiple event in log file
> [@steffens](#):
>
> With multiple events, are they put into separate arrays, or all in the top-level array?
It is separate arrays.
> [@steffens](#):
>
> Is the log pretty printed or not?
No. It look like raw look above.
Thanks you!
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [August 14, 2017, 12:59pm UTC](https://discuss.elastic.co/t/how-to-read-log-json-array-format-with-filebeat/96220/7 "2017-08-14T12:59:02Z")
Do you have a sample log (multiple events)? I have a hard time tell how logs look like and how processing could be done.
---
### Author:  [@tatdat](https://discuss.elastic.co/u/tatdat)
#### Post date: [August 15, 2017, 12:49am UTC](https://discuss.elastic.co/t/how-to-read-log-json-array-format-with-filebeat/96220/8 "2017-08-15T00:49:06Z")
Sorry, i checked my log, it have only one array in one file and have many file 🙂
> [@tatdat](#):
>
> With multiple events, are they put into separate arrays, or all in the top-level array?
All in the top-level array. Like this. It have multiple events.
[https://pastebin.com/raw/AXh80s3m](https://pastebin.com/raw/AXh80s3m)
---
### Author:  [@tatdat](https://discuss.elastic.co/u/tatdat)
#### Post date: [August 21, 2017, 6:58am UTC](https://discuss.elastic.co/t/how-to-read-log-json-array-format-with-filebeat/96220/9 "2017-08-21T06:58:39Z")
@steffens Can u help me solve my problem? Thanks !
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [August 21, 2017, 3:38pm UTC](https://discuss.elastic.co/t/how-to-read-log-json-array-format-with-filebeat/96220/10 "2017-08-21T15:38:45Z")
I checked a hexdump of your logfile. The problem is not multiline. Actually you don't need multiline for this log file. I see some interesting things in your log file:
1. No newline character at the end. The newline is required by filebeat to determine a log line is complete.
2. Some wird binary sequence (non-printable characters?) At the end of your file (last 10 bytes)
3. Not sure if I'm missing some fonts, but I can not correctly see some of the embedded strings with default UTF-8 encoding in my editor. Seems like windows path being included as is, in UTF-16 encoding. E.g. is this path correct? `\\MuÌ£c Ä‘ã xoÌ` (looks like some generated path to store some actual mail content in)?
I guess these files are written at once and never been touched afterwards?
---
### Author:  [@tatdat](https://discuss.elastic.co/u/tatdat)
#### Post date: [August 23, 2017, 2:36am UTC](https://discuss.elastic.co/t/how-to-read-log-json-array-format-with-filebeat/96220/11 "2017-08-23T02:36:42Z")
Thank @steffens. I sent to you my answer via inbox messagage.
> Not sure if I’m missing some fonts, but I can not correctly see some of the embedded strings with default UTF-8 encoding in my editor. Seems like windows path being included as is, in UTF-16 encoding. E.g. is this path correct? \MuÌ£c Ä‘ã xoÌ (looks like some generated path to store some actual mail content in)?
This path is correctly. It generate by O365. I dont know that why. Maybe Log )365 dont support my language.
> I guess these files are written at once and never been touched afterwards?
Yeah, i dont touch anything.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 20, 2017, 2:36am UTC](https://discuss.elastic.co/t/how-to-read-log-json-array-format-with-filebeat/96220/12 "2017-09-20T02:36:43Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.