Trying to parse a multine array of json objects with logstash split and json

Elastic Stack Logstash
1

Hi all,

I'll preface this by saying that I'm still very new to ELK so I'm only doing what I think is the right way to do things, if there are any mistakes in my process feel free to point them out. Anyways, here's the jist of the problem, I have an application that generates an array of json objects of a summary on various vms (short version shown below) and just want logstash to parse each element in the array as an individual entry with each key becoming a field key and each value becoming a field value in that entry.

[
{
    "total_ram":  2681,
    "dc_count":  1,
    "description":  "VMWare Capacity \u0026 Performance Report for GBCDFVCEP011",
    "server_name":  "GBCDFVCEP011",
    "version":  "6.5.0 Build 9451637",
    "total_cpu":  292,
    "vtype":  "server",
    "template_count":  0,
    "vms_count":  98,
    "consolidation_ratio":  "14:1",
    "vm":  {
               "win":  90,
               "other":  8,
               "total":  98,
               "nix":  0
           },
    "recource_pool_count":  24,
    "timestamp":  1557484988,
    "cluster_count":  1,
    "cpu_count":  112,
    "host_count":  7
},
{
    "model":  "Dell Inc. PowerEdge R720",
    "cur_cpu_usage":  0.49,
    "avg_mem_usage":  0.55,
    "version":  "6.0.0 Build 3620759",
    "avg_cpu_usage":  0.3,
    "server_name":  "GBCDFVCEP011",
    "uptime":  231,
    "total_ram":  384,
    "host":  "gbcdfesxp016.atradiusnet.com",
    "cpu_count":  16,
    "cur_mem_usage":  0.54,
    "vtype":  "host"
},
{
    "model":  "Dell Inc. PowerEdge R720",
    "cur_cpu_usage":  0.64,
    "avg_mem_usage":  0.56,
    "version":  "6.0.0 Build 3620759",
    "avg_cpu_usage":  0.4,
    "server_name":  "GBCDFVCEP011",
    "uptime":  230,
    "total_ram":  384,
    "host":  "gbcdfesxp017.atradiusnet.com",
    "cpu_count":  16,
    "cur_mem_usage":  0.58,
    "vtype":  "host"
},

...

]

And these are the filbeat config and logstash filter:
logstash

json {
source => "message"
  }

  split {
field => "message"
  }

filebeat

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
- /var/log/*.log
- /var/log/*.json
#- c:\programdata\elasticsearch\logs\*
  fields:
proj_index: 'windows-esxi'

  multiline.pattern: '^[[:space:]]+\{'
  multiline.negate: true
  multiline.match:  after
  json.keys_under_root: true
  json.message_key: message

With this configuration it manages to get all the lines in the file but it doesn't manage to actually parse them. Instead of parsing each element in the array as an entry it just put the entire message as the message attribute of a single entry. Any help is appreciated.

2

Are you trying to get filebeat to parse the json or do you want logstash to do that?

3

I'd prefer if logstash did the parsing.

4

If you want logstash to do it then if the JSON is an array you must specify the target option on the json filter.

5

I see, thank you. But how do I make it so that the array is split into individual log entries? The split function doesn't seem to work, or at least it doesn't work in the way I understood it should.

6

You can do it using

    json { source => "message" target => "someField" }
    split { field => "someField" }
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.