Visualising Nested Objects in Kibana

weiyentan · November 7, 2020, 10:30pm

Hi,

I am exploring using ELK and imported in this json document:

{
  "_index": "system-inventory-[version]}-2020.11.07",
  "_type": "_doc",
  "_id": "O0X1o3UBD7PZ5Lh--gIu",
  "_version": 1,
  "_score": null,
  "_source": {
    "host": "10.42.1.1",
    "osversion": "Microsoft Windows 10 Enterprise",
    "computername": "WIN10CLIENT",
    "software": [
      {
        "DisplayName": "7-Zip 20.02 alpha (x64)",
        "DisplayVersion": "20.02 alpha"
      },
      {
        "DisplayName": "Git version 2.27.0",
        "DisplayVersion": "2.27.0"
      },
      {
        "DisplayName": "Mozilla Firefox 80.0.1 (x64 en-GB)",
        "DisplayVersion": "80.0.1"
      },
      {
        "DisplayName": "Mozilla Maintenance Service",
        "DisplayVersion": "80.0.1"
      },
      {
        "DisplayName": "SAPIEN PowerShell Studio 2020",
        "DisplayVersion": "5.7.181.0"
      },
      {
        "DisplayName": "SAPIEN Updates",
        "DisplayVersion": "1.1.37.0"
      },
      {
        "DisplayName": "SAPIEN ScriptMerge 2020",
        "DisplayVersion": "1.4.83.0"
      },
      {
        "DisplayName": "PowerShell 7-x64",
        "DisplayVersion": "7.0.3.0"
      },
      {
        "DisplayName": "MiniTool Partition Wizard Free 12",
        "DisplayVersion": null
      },
      {
        "DisplayName": "VMware Tools",
        "DisplayVersion": "10.2.1.8267844"
      },
      {
        "DisplayName": "SQL Server Management Studio",
        "DisplayVersion": "15.0.18338.0"
      },
      {
        "DisplayName": "SQL Server Management Studio for Analysis Services",
        "DisplayVersion": "15.0.18338.0"
      },
      {
        "DisplayName": "Microsoft Visual C++ 2019 X64 Additional Runtime - 14.21.27702",
        "DisplayVersion": "14.21.27702"
      },
      {
        "DisplayName": "VNC Viewer 5.2.3",
        "DisplayVersion": "5.2.3"
      },
      {
        "DisplayName": "Microsoft .NET Core Runtime - 3.1.3 (x64)",
        "DisplayVersion": "24.76.28628"
      },
      {
        "DisplayName": "Microsoft .NET Core Host - 3.1.3 (x64)",
        "DisplayVersion": "24.76.28628"
      },
      {
        "DisplayName": "ScriptMerge 2020",
        "DisplayVersion": "1.4.83.0"
      },
      {
        "DisplayName": "Beats metricbeat 7.9.3 (x86_64)",
        "DisplayVersion": "7.9.3"
      },
      {
        "DisplayName": "TightVNC",
        "DisplayVersion": "2.8.27.0"
      },
      {
        "DisplayName": "Microsoft Visual C++ 2008 Redistributable - x64 9.0.30729.6161",
        "DisplayVersion": "9.0.30729.6161"
      },
      {
        "DisplayName": "Update for Windows 10 for x64-based Systems (KB4023057)",
        "DisplayVersion": "2.67.0.0"
      },
      {
        "DisplayName": "SQL Server Management Studio",
        "DisplayVersion": "15.0.18338.0"
      },
      {
        "DisplayName": "Duo Authentication for Windows Logon x64",
        "DisplayVersion": "4.1.0.283"
      },
      {
        "DisplayName": "Microsoft OLE DB Driver for SQL Server",
        "DisplayVersion": "18.3.0.0"
      },
      {
        "DisplayName": "SSMS Post Install Tasks",
        "DisplayVersion": "15.0.18338.0"
      },
      {
        "DisplayName": "Microsoft Windows Desktop Runtime - 3.1.3 (x64)",
        "DisplayVersion": "24.76.28628"
      },
      {
        "DisplayName": "Beats filebeat 7.9.3 (x86_64)",
        "DisplayVersion": "7.9.3"
      },
      {
        "DisplayName": "Microsoft Silverlight",
        "DisplayVersion": "5.1.50918.0"
      },
      {
        "DisplayName": "PowerShell Studio 2020",
        "DisplayVersion": "5.7.181.0"
      },
      {
        "DisplayName": "SQL Server Management Studio for Reporting Services",
        "DisplayVersion": "15.0.18338.0"
      },
      {
        "DisplayName": "Microsoft SQL Server 2012 Native Client ",
        "DisplayVersion": "11.4.7462.6"
      },
      {
        "DisplayName": "Microsoft Visual Studio Tools for Applications 2017 x64 Hosting Support",
        "DisplayVersion": "15.0.27520"
      },
      {
        "DisplayName": "SAPIEN Updates",
        "DisplayVersion": "1.1.37.0"
      },
      {
        "DisplayName": "Update for Windows 10 for x64-based Systems (KB4480730)",
        "DisplayVersion": "2.55.0.0"
      },
      {
        "DisplayName": "Microsoft Analysis Services OLE DB Provider",
        "DisplayVersion": "15.0.2000.20"
      },
      {
        "DisplayName": "Microsoft .NET Core Host FX Resolver - 3.1.3 (x64)",
        "DisplayVersion": "24.76.28628"
      },
      {
        "DisplayName": "Microsoft ODBC Driver 17 for SQL Server",
        "DisplayVersion": "17.5.1.1"
      },
      {
        "DisplayName": "Microsoft Visual C++ 2019 X64 Minimum Runtime - 14.21.27702",
        "DisplayVersion": "14.21.27702"
      },
      {
        "DisplayName": "Adobe Flash Player 32 NPAPI",
        "DisplayVersion": "32.0.0.445"
      },
      {
        "DisplayName": "Adobe Flash Player 32 PPAPI",
        "DisplayVersion": "32.0.0.445"
      },
      {
        "DisplayName": "Microsoft Edge",
        "DisplayVersion": "86.0.622.63"
      },
      {
        "DisplayName": "Microsoft Edge Update",
        "DisplayVersion": "1.3.137.99"
      },
      {
        "DisplayName": "Microsoft Help Viewer 2.3",
        "DisplayVersion": "2.3.28107"
      },
      {
        "DisplayName": "MiniTool ShadowMaker Free Edition",
        "DisplayVersion": "3.2"
      },
      {
        "DisplayName": "No-IP DUC",
        "DisplayVersion": "4.1.1"
      },
      {
        "DisplayName": "WinSCP 5.17.7",
        "DisplayVersion": "5.17.7"
      },
      {
        "DisplayName": "Microsoft SQL Server Management Studio - 18.6",
        "DisplayVersion": "15.0.18338.0"
      },
      {
        "DisplayName": "Microsoft Visual C++ 2013 x86 Minimum Runtime - 12.0.21005",
        "DisplayVersion": "12.0.21005"
      },
      {
        "DisplayName": "Microsoft Visual C++ 2019 X86 Minimum Runtime - 14.21.27702",
        "DisplayVersion": "14.21.27702"
      },
      {
        "DisplayName": "Python 3.8.4 Utility Scripts (32-bit)",
        "DisplayVersion": "3.8.4150.0"
      },
      {
        "DisplayName": "Microsoft Visual C++ 2019 X86 Additional Runtime - 14.21.27702",
        "DisplayVersion": "14.21.27702"
      },
      {
        "DisplayName": "Microsoft Visual C++ 2015-2019 Redistributable (x86) - 14.21.27702",
        "DisplayVersion": "14.21.27702.2"
      },
      {
        "DisplayName": "Python 3.8.4 Documentation (32-bit)",
        "DisplayVersion": "3.8.4150.0"
      },
      {
        "DisplayName": "Microsoft Analysis Services OLE DB Provider",
        "DisplayVersion": "15.0.2000.20"
      },
      {
        "DisplayName": "Python 3.8.4 Executables (32-bit)",
        "DisplayVersion": "3.8.4150.0"
      },
      {
        "DisplayName": "Python Launcher",
        "DisplayVersion": "3.8.7133.0"
      },
      {
        "DisplayName": "Microsoft Visual Studio Tools for Applications 2017 x86 Hosting Support",
        "DisplayVersion": "15.0.27520"
      },
      {
        "DisplayName": "Microsoft Visual C++ 2008 Redistributable - x86 9.0.30729.6161",
        "DisplayVersion": "9.0.30729.6161"
      },
      {
        "DisplayName": "Python 3.8.4 Tcl/Tk Support (32-bit)",
        "DisplayVersion": "3.8.4150.0"
      },
      {
        "DisplayName": "Python 3.8.4 Standard Library (32-bit)",
        "DisplayVersion": "3.8.4150.0"
      },
      {
        "DisplayName": "Visual Studio 2017 Isolated Shell for SSMS",
        "DisplayVersion": "15.0.28307.421"
      },
      {
        "DisplayName": "Python 3.8.4 Core Interpreter (32-bit)",
        "DisplayVersion": "3.8.4150.0"
      },
      {
        "DisplayName": "Python 3.8.4 Add to Path (32-bit)",
        "DisplayVersion": "3.8.4150.0"
      },
      {
        "DisplayName": "MindMaster(Build 8.0.4.115)",
        "DisplayVersion": "8.0.4.115"
      },
      {
        "DisplayName": "Python 3.8.4 pip Bootstrap (32-bit)",
        "DisplayVersion": "3.8.4150.0"
      },
      {
        "DisplayName": "Microsoft Help Viewer 2.3",
        "DisplayVersion": "2.3.28107"
      },
      {
        "DisplayName": "Python 3.8.4 Test Suite (32-bit)",
        "DisplayVersion": "3.8.4150.0"
      },
      {
        "DisplayName": "Integration Services",
        "DisplayVersion": "15.0.2000.118"
      },
      {
        "DisplayName": "Python 3.8.4 Development Libraries (32-bit)",
        "DisplayVersion": "3.8.4150.0"
      },
      {
        "DisplayName": "Microsoft Visual C++ 2015-2019 Redistributable (x64) - 14.21.27702",
        "DisplayVersion": "14.21.27702.2"
      },
      {
        "DisplayName": "Microsoft Visual C++ 2013 Redistributable (x86) - 12.0.30501",
        "DisplayVersion": "12.0.30501.0"
      },
      {
        "DisplayName": "Microsoft Windows Desktop Runtime - 3.1.3 (x64)",
        "DisplayVersion": "3.1.3.28628"
      },
      {
        "DisplayName": "Microsoft Visual Studio Tools for Applications 2017",
        "DisplayVersion": "15.0.27520"
      },
      {
        "DisplayName": "Microsoft Visual C++ 2013 x86 Additional Runtime - 12.0.21005",
        "DisplayVersion": "12.0.21005"
      }
    ],
    "@version": "1",
    "headers": {
      "http_host": "192.168.1.105:6000",
      "request_path": "/",
      "connection": "Keep-Alive",
      "request_method": "PUT",
      "http_user_agent": "Mozilla/5.0 (Windows NT; Windows NT 10.0; en-NZ) WindowsPowerShell/5.1.18362.752",
      "content_type": "application/json",
      "http_accept": null,
      "http_version": "HTTP/1.1",
      "content_length": "14175"
    },
    "@timestamp": "2020-11-07T18:26:47.570Z",
    "Patchesneeded": 0
  },
  "fields": {
    "@timestamp": [
      "2020-11-07T18:26:47.570Z"
    ]
  },
  "sort": [
    1604773607570
  ]
}

This json document is imported into the ELK through logstash.

In kibana what I would like is to list the software and match it to a computer.

However when I try to match the contents of software to a computer in Kibana, there is no obvious way of doing this. I ama brand new use of ELK, so I am still learning.

I read that Kibana doesnt really do nested objects and I understand that. I was wondering if there was any other way that we can pull out that nested object so that I can achieve what i want?

I will be gratefyul for any guidance. Many thanks.

Wei-Yen Tan

wylie · November 9, 2020, 4:03pm

Instead of organizing your data as one document with multiple nested documents, can you organize it as multiple documents for each child? This is usually done with the logstash split filter

weiyentan · November 17, 2020, 8:22am

I never really thought of that. I will have a look to see how that works. How would i then piece together in Kibana ? Thank you

wylie · November 17, 2020, 4:10pm

The Elastic stack doesn't really allow correlation across documents, except by using filtering. You can use filters to collect all the documents for a specific filter. For example, if you search for "host": "10.42.1.1", you would see the 20+ documents representing each piece of software above.

If you need two different types of analysis on this data, you could benefit from having 2 logstash pipelines. One could split the data, and the other could keep it structured the way you showed above.

weiyentan · November 20, 2020, 7:13am

Thank you @wylie ,

I think you are right. This is my first time using ELK so I am exploring the various different ways of using this and trying to work with the way its supposed to work.

I am actually using scripts to generate json documents and pushing that through an http in plugin of log stash. So I control the input.

So based on what you said is it better to create individual document (pieces of information, the flatter it is the better) and then use filtering and dash boards to bring the data together? In other words to normalize the data?

If so I will rewrite my pieces of automation work to accomodate for that.

wylie · November 20, 2020, 3:33pm

I would actually describe it as denormalizing: you are encouraged to copy data into a structure that makes it easy to analyze. It seems like you wanted to analyze your data at 2 levels of the hierarchy, so I suggested you create 2 different representations of the data:

To analyze "software", each piece of software should have its own document using the split filter
To visualize "computernames" or "hosts", you can use the structure you originally suggested.

This is mostly because Elasticsearch doesn't support joins in the SQL sense, so you need to structure your documents around what you need to index.

weiyentan · November 23, 2020, 7:17am

Thank you @wylie for the reply. After reading through the documentation I realised that I can use split on a field. In that same split would I be able to add a field to the 'copy'?
Example:

input {
  http {
    port => 6000 # default: 8080
    codec => json
  }
}
   filter {
    split {
       field => "software"
    }
   split {
      field => "patchesneeded"
  }

  split {
    add_field => {"computername" => "%{computername}"
  }
  }
output {
elasticsearch {
hosts => ["http://elasticsearch:9200"]
index => "system-inventory-[version]}-%{+YYYY.MM.dd}"
}
stdout {
codec => rubydebug
}
}

It looks like when I ran this inside my logstash configmap the Pod shut down.

It looks like its failing on this:

  split {
    add_field => {"computername" => "%{computername}"
  }

I have a field called computername and would like to add it to the split. Is this how it works?

wylie · November 23, 2020, 6:59pm

I think you're asking a fine question, but I don't know how to answer it. Logstash has a separate forum if you want to ask for help configuring your logstash patterns, you can ask for help if you describe the input, logstash transform, and desired output. I

system · December 21, 2020, 6:59pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Data is showing wrong when i add some nested fields Kibana	2	261	March 22, 2021
Nested objects in Elasticsearch and Kibana Logstash	2	314	February 4, 2019
Parsing Nested JSONS Kibana	2	243	August 6, 2019
Kibana Visualisation on Nested Field Objects Kibana	2	250	March 20, 2019
Nested objects, to flatten or split? Logstash	2	533	December 24, 2019

Visualising Nested Objects in Kibana

Related Topics