Problem getting multiline json document into ES

[ajhstn]

Hi, i am producing a valid json document from get-aduser username | convertto-json | out-file file.json

I am then sending that file to ES using filebeat, however in ES it is ingesting the document, one document per line, but because the json document is multiline it's obviously not what i need.

How do i get the entire json object as a single document in ES rather then a document per line?

Follow on question would be, say i want to do get-aduser -filter * and output that to a json file, and send that in, does the same rule apply?

file.json

{
"GivenName":  "Joe",
"Surname":  "Bloggs",
"UserPrincipalName":  "job@domain.com",
"Enabled":  true,
"SamAccountName":  "job",
"SID":  {
"BinaryLength":  28,
"AccountDomainSid":  {
"BinaryLength":  24,
"AccountDomainSid":  "S-1-5-21-nnnn-nnnn-nnnn",
"Value":  "S-1-5-21-nnnn-nnnn-nnnn"
},
"Value":  "S-1-5-21-nnnn-nnnn-nnnn-nnnn"
},
"DistinguishedName":  "CN=Joe Bloggs,DC=domain,DC=com",
"Name":  "Joe Bloggs",
"ObjectClass":  "user",
"ObjectGuid":  "nnnn-nnnn-nnnn-nnnn-nnnn"
 }

Filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - "D:\\dev\\ADUsers.json"
  processors:
     - decode_json_fields:
       fields: ['message']

[forloop (Russ Cam)]

@ajhstn

get-aduser username | convertto-json | out-file file.json

I think you need to emit compressed JSON to output the user JSON object on one line, which is what filebeat expects

Get-AdUser username | ConvertTo-Json -Compress | Out-File file.json

If you wanted to get multiple users into a file, one JSON object per line

Get-AdUser -Filter * | For-EachObject { $_ | ConvertTo-Json -Compress } | Out-File file.json

[ajhstn]

This kind of works, i now get the whole JSON doc in the "message" field, however it isn't decoded into seperate JSON objects?

[ajhstn]

Rather then reading JSON from a file with Filebeat, i was able to POST the JSON object directly to my index, which does seem to work and automatically, creates the itemised fields and templates for me.

Question: In a foreach loop through 100s, 1000s of users, doing a POST each iteration, is that a supported or recommended way, or should i persist with Filebeat?

[ajhstn]

Here is my PowerShell code

$u = Get-ADUser bob | Select-Object DistinguishedName, Enabled, GivenName, ObjectClass, ObjectGUID, SamAccountName, Surname, UserPrincipalName

@{ timestamp = (Get-Date);  message= $u } | ConvertTo-Json -Compress | Out-File .\adusers.json -Force

Here is my filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - "D:\\dev\\ADUsers.json"
  processors:
  - decode_json_fields:
      fields: ['message']

index: "adusers"

I just can't quite get it right.

Side note: I am trying to create a new index also called adusers but the data keeps going into the filebeat index instead.

[forloop (Russ Cam)]

@ajhstn

Question: In a foreach loop through 100s, 1000s of users, doing a POST each iteration, is that a supported or recommended way, or should i persist with Filebeat?

It's preferable to send multiple documents in one HTTP request, using the _bulk API, which is what filebeat does.

@ajhstn

$u = Get-ADUser bob | Select-Object DistinguishedName, Enabled, GivenName, ObjectClass, ObjectGUID, SamAccountName, Surname, UserPrincipalName @{ timestamp = (Get-Date); message= $u } | ConvertTo-Json -Compress | Out-File .\adusers.json -Force

I think what you want, is to append the timestamp to the set of available properties selected from the ADUser object, then serialize this to JSON? If so, you can use a calculated property for this. Something like

# guess you want the same timestamp for all retrieved objects?
$timestamp = (Get-Date).ToUniversalTime().ToString("u")

Get-ADUser bob | 
Select-Object DistinguishedName, Enabled, GivenName, `
    ObjectClass, ObjectGUID, SamAccountName, Surname, `
    UserPrincipalName, @{Name = '@timestamp'; Expression = { $timestamp }} | 
ForEach-Object { $_ | ConvertTo-Json -Compress } |
Out-File .\adusers.json