# Problem getting multiline json document into ES

**URL:** <https://discuss.elastic.co/t/problem-getting-multiline-json-document-into-es/194350>\
**Category:** Elasticsearch\
**Created:** [August 8, 2019, 4:25am UTC](https://discuss.elastic.co/t/problem-getting-multiline-json-document-into-es/194350 "2019-08-08T04:25:09Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![ajhstn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajhstn/32/24629_2.png) [@ajhstn](https://discuss.elastic.co/u/ajhstn)\
**Post date:** [August 8, 2019, 4:25am UTC](https://discuss.elastic.co/t/problem-getting-multiline-json-document-into-es/194350/1 "2019-08-08T04:25:09Z")

</div>

Hi, i am producing a valid json document from `get-aduser username | convertto-json | out-file file.json`

I am then sending that file to ES using filebeat, however in ES it is ingesting the document, one document per line, but because the json document is multiline it's obviously not what i need.

How do i get the entire json object as a single document in ES rather then a document per line?

Follow on question would be, say i want to do `get-aduser -filter *` and output that to a json file, and send that in, does the same rule apply?

file.json

```
{
"GivenName": "Joe",
"Surname": "Bloggs",
"UserPrincipalName": "job@domain.com",
"Enabled": true,
"SamAccountName": "job",
"SID": {
"BinaryLength": 28,
"AccountDomainSid": {
"BinaryLength": 24,
"AccountDomainSid": "S-1-5-21-nnnn-nnnn-nnnn",
"Value": "S-1-5-21-nnnn-nnnn-nnnn"
},
"Value": "S-1-5-21-nnnn-nnnn-nnnn-nnnn"
},
"DistinguishedName": "CN=Joe Bloggs,DC=domain,DC=com",
"Name": "Joe Bloggs",
"ObjectClass": "user",
"ObjectGuid": "nnnn-nnnn-nnnn-nnnn-nnnn"
 }

```

Filebeat.yml

```
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - "D:\\dev\\ADUsers.json"
  processors:
     - decode_json_fields:
       fields: ['message']
```

---

<div class="post-metadata">

**Author:** ![forloop](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forloop/32/9021_2.png) [@forloop](https://discuss.elastic.co/u/forloop)\
**Post date:** [August 8, 2019, 4:48am UTC](https://discuss.elastic.co/t/problem-getting-multiline-json-document-into-es/194350/2 "2019-08-08T04:48:14Z")

</div>

> [@ajhstn](#):
>
> get-aduser username | convertto-json | out-file file.json

I think you need to emit compressed JSON to output the user JSON object on one line, which is what [filebeat expects](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#filebeat-input-log-config-json)

```powershell
Get-AdUser username | ConvertTo-Json -Compress | Out-File file.json

```

If you wanted to get multiple users into a file, one JSON object per line

```powershell
Get-AdUser -Filter * | For-EachObject { $_ | ConvertTo-Json -Compress } | Out-File file.json

```

---

<div class="post-metadata">

**Author:** ![ajhstn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajhstn/32/24629_2.png) [@ajhstn](https://discuss.elastic.co/u/ajhstn)\
**Post date:** [August 8, 2019, 6:08am UTC](https://discuss.elastic.co/t/problem-getting-multiline-json-document-into-es/194350/3 "2019-08-08T06:08:13Z")

</div>

This kind of works, i now get the whole JSON doc in the "message" field, however it isn't decoded into seperate JSON objects?

---

<div class="post-metadata">

**Author:** ![ajhstn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajhstn/32/24629_2.png) [@ajhstn](https://discuss.elastic.co/u/ajhstn)\
**Post date:** [August 8, 2019, 6:12am UTC](https://discuss.elastic.co/t/problem-getting-multiline-json-document-into-es/194350/4 "2019-08-08T06:12:56Z")

</div>

Rather then reading JSON from a file with Filebeat, i was able to POST the JSON object directly to my index, which does seem to work and automatically, creates the itemised fields and templates for me.

Question: In a foreach loop through 100s, 1000s of users, doing a POST each iteration, is that a supported or recommended way, or should i persist with Filebeat?

---

<div class="post-metadata">

**Author:** ![ajhstn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajhstn/32/24629_2.png) [@ajhstn](https://discuss.elastic.co/u/ajhstn)\
**Post date:** [August 8, 2019, 6:22am UTC](https://discuss.elastic.co/t/problem-getting-multiline-json-document-into-es/194350/5 "2019-08-08T06:22:17Z")

</div>

Here is my PowerShell code

```
$u = Get-ADUser bob | Select-Object DistinguishedName, Enabled, GivenName, ObjectClass, ObjectGUID, SamAccountName, Surname, UserPrincipalName

@{ timestamp = (Get-Date); message= $u } | ConvertTo-Json -Compress | Out-File .\adusers.json -Force

```

Here is my `filebeat.yml`

```
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - "D:\\dev\\ADUsers.json"
  processors:
  - decode_json_fields:
      fields: ['message']

index: "adusers"

```

I just can't quite get it right.

Side note: I am trying to create a new index also called `adusers` but the data keeps going into the `filebeat` index instead.

---

<div class="post-metadata">

**Author:** ![forloop](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/forloop/32/9021_2.png) [@forloop](https://discuss.elastic.co/u/forloop)\
**Post date:** [August 8, 2019, 6:56am UTC](https://discuss.elastic.co/t/problem-getting-multiline-json-document-into-es/194350/6 "2019-08-08T06:56:04Z")

</div>

> [@ajhstn](#):
>
> Question: In a foreach loop through 100s, 1000s of users, doing a POST each iteration, is that a supported or recommended way, or should i persist with Filebeat?

It's preferable to send multiple documents in one HTTP request, using the [`_bulk` API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html), which is what filebeat does.

> [@ajhstn](#):
>
> ```auto
> $u = Get-ADUser bob | Select-Object DistinguishedName, Enabled, GivenName, ObjectClass, ObjectGUID, SamAccountName, Surname, UserPrincipalName @{ timestamp = (Get-Date); message= $u } | ConvertTo-Json -Compress | Out-File .\adusers.json -Force
> 
> ```

I think what you want, is to append the timestamp to the set of available properties selected from the `ADUser` object, then serialize this to JSON? If so, you can use a calculated property for this. Something like

```ps
# guess you want the same timestamp for all retrieved objects?
$timestamp = (Get-Date).ToUniversalTime().ToString("u")

Get-ADUser bob | 
Select-Object DistinguishedName, Enabled, GivenName, `
    ObjectClass, ObjectGUID, SamAccountName, Surname, `
    UserPrincipalName, @{Name = '@timestamp'; Expression = { $timestamp }} | 
ForEach-Object { $_ | ConvertTo-Json -Compress } |
Out-File .\adusers.json

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 5, 2019, 6:56am UTC](https://discuss.elastic.co/t/problem-getting-multiline-json-document-into-es/194350/7 "2019-09-05T06:56:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.