Problem getting multiline json document into ES

Elastic Stack Elasticsearch
1

Hi, i am producing a valid json document from get-aduser username | convertto-json | out-file file.json

I am then sending that file to ES using filebeat, however in ES it is ingesting the document, one document per line, but because the json document is multiline it's obviously not what i need.

How do i get the entire json object as a single document in ES rather then a document per line?

Follow on question would be, say i want to do get-aduser -filter * and output that to a json file, and send that in, does the same rule apply?

file.json

{
"GivenName":  "Joe",
"Surname":  "Bloggs",
"UserPrincipalName":  "job@domain.com",
"Enabled":  true,
"SamAccountName":  "job",
"SID":  {
"BinaryLength":  28,
"AccountDomainSid":  {
"BinaryLength":  24,
"AccountDomainSid":  "S-1-5-21-nnnn-nnnn-nnnn",
"Value":  "S-1-5-21-nnnn-nnnn-nnnn"
},
"Value":  "S-1-5-21-nnnn-nnnn-nnnn-nnnn"
},
"DistinguishedName":  "CN=Joe Bloggs,DC=domain,DC=com",
"Name":  "Joe Bloggs",
"ObjectClass":  "user",
"ObjectGuid":  "nnnn-nnnn-nnnn-nnnn-nnnn"
 }

Filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - "D:\\dev\\ADUsers.json"
  processors:
     - decode_json_fields:
       fields: ['message']
2

I think you need to emit compressed JSON to output the user JSON object on one line, which is what filebeat expects

Get-AdUser username | ConvertTo-Json -Compress | Out-File file.json

If you wanted to get multiple users into a file, one JSON object per line

Get-AdUser -Filter * | For-EachObject { $_ | ConvertTo-Json -Compress } | Out-File file.json
3

This kind of works, i now get the whole JSON doc in the "message" field, however it isn't decoded into seperate JSON objects?

4

Rather then reading JSON from a file with Filebeat, i was able to POST the JSON object directly to my index, which does seem to work and automatically, creates the itemised fields and templates for me.

Question: In a foreach loop through 100s, 1000s of users, doing a POST each iteration, is that a supported or recommended way, or should i persist with Filebeat?

5

Here is my PowerShell code

$u = Get-ADUser bob | Select-Object DistinguishedName, Enabled, GivenName, ObjectClass, ObjectGUID, SamAccountName, Surname, UserPrincipalName

@{ timestamp = (Get-Date);  message= $u } | ConvertTo-Json -Compress | Out-File .\adusers.json -Force

Here is my filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - "D:\\dev\\ADUsers.json"
  processors:
  - decode_json_fields:
      fields: ['message']

index: "adusers"

I just can't quite get it right.

Side note: I am trying to create a new index also called adusers but the data keeps going into the filebeat index instead.

6

It's preferable to send multiple documents in one HTTP request, using the _bulk API, which is what filebeat does.

I think what you want, is to append the timestamp to the set of available properties selected from the ADUser object, then serialize this to JSON? If so, you can use a calculated property for this. Something like

# guess you want the same timestamp for all retrieved objects?
$timestamp = (Get-Date).ToUniversalTime().ToString("u")

Get-ADUser bob | 
Select-Object DistinguishedName, Enabled, GivenName, `
    ObjectClass, ObjectGUID, SamAccountName, Surname, `
    UserPrincipalName, @{Name = '@timestamp'; Expression = { $timestamp }} | 
ForEach-Object { $_ | ConvertTo-Json -Compress } |
Out-File .\adusers.json
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.