Recording CVE data in Elastic and displaying with Kibana

[tomdell]

Hi all, I haven't been able to find anyone else really doing this, and after a week of playing around I'm still pretty lost.

Currently my goal is to ingest vulnerability data into ELK, and then use Kibana to see the data, however the issue I am running into is that there is a lot of nested data, which I can't seem to extract and display.

One of the original solutions was to flatten the documents as much as possible, however due to the nature of vulnerability data, it result in too many documents, as each vulnerability has multiple updates, multiple affects products, and multiple external source links. Flattening the documents results in over 200 (sometimes way more) documents per one vulnerability;

• affected product 1, history 1, source 1
• affected product 1, history 1, source 2
• affected product 1, history 2, source 1
• affected product 1, history 2, source 2

When vulnerabilities affect 20+ products and have 20 updates, you can imagine how many documents would be created.

The best way I found to get around this was to nest the data, however this has caused problems of it's own, it seems that I am unable to use of the data in a 'nested' field in any visualisations.

The plan is to create a few different dashboards, one of them being a CVE Search function, where a user would enter a CVE and be returned with all the information, such as severity affected products, and a list of URLs, which is a problem as both of those have information stored in a nested field.

.

Pulling the CVE, and description is easy, the problem is getting the information in the references field, as it's been defined as a nested field.

I was wondering if anyone had any ideas on how I could go about this? It's starting to seem like ELK may not be the best place for this type of data.

[joshdover]

First, I have to say I think what you're trying to build is super great! If you ever release this publicly, please drop me a note so I can play around with it

It appears that you are running into one of Kibana's limitations, which is its limited support for doing aggregations on nested arrays. The good news is this is something the team is considering improving soon the bad news is this doesn't solve your problem today.

There are however some other options:

• I would start with exploring the Enhanced Table Plugin. This plugin is one of the most popular community plugins, is well maintained, and feature packed. It does not yet support 7.6, but that is actively in progress. If you can use 7.5 (or lower) in the meantime, this may do the things you need.
• While Kibana may be lacking the features you need, I think Elasticsearch itself is still fully capable of doing what you need and, if practical, you could build a custom UI on top of it. I would suggest checking out our Search UI library for building a search-based UI in React on top of Elasticsearch. This should give you a good starting point for building something custom.

I hope this was helpful and I look forward to the day Kibana has better out-of-the-box support for this type of data!

[tomdell]

Thank you for the reply Josh, I will definitely look into the Enhanced Table Plugin. I will also look into other front ends, or even building something. Now that I know 100% that's it's a limitation in Kibana I can keep the rest of the stack and start looking for something else that could work for me.

Thanks again

[tomdell]

Hi @joshdover, is there a way to extract fields from within the nested field in Discovery in some kind of for loop or such? If that works and we can extract all of the data in the 'URL' field, then we can save that as a search, and then display that search in a Dashboard?

It would be good to iterate through the "CVE_Items.cve.references.reference_data" extracting the "name" and the "URL" data and displaying it like below;

Where $name and $URL are the "name" and "URL" fields under "CVE_Items.cve.references.reference_data".

Because if Discovery can do that, the saved search function in Dashboard could replace the Data Table visualisation.

[joshdover]

The only way I know how to do this is by using scripted fields which allow you to generate new columns based on data in other fields. A couple caveats to note: you cannot search or aggregate over scripted fields, but you can easily display them.

Using this you could loop over the items in a nested array and build a single string with all the data embedded.

[tomdell]

Ahh, that is a bit of a deal breaker, as affected products are in one of those nested fields, and we would need to be able to search by product.

That's a shame, I thought there might have been a way to iterate through the data in a nested field using some complex search query.

[joshdover]

One option would be to use either an Ingest Pipeline or Enrich processor in Elasticsearch to generate these computed fields when the documents are ingested or updated in Elasticsearch.

One really nice thing about the Enrich processor is that you can lookup data from other indices to use when adding new fields to the documents. You can also re-run the processor on existing documents to update any stale data that may have changed in other indices.

If you go with either of these routes you'll be able to search on these fields.

[tomdell]

It honestly sounds like the best idea is to wait until Kibana has support for nested fields, the issue received a update 14 days ago, so hopefully that's a sign of good things to come.