Kibana List API Questions

[willemdh]

Hello all,

We really like the Kibana List API. it enables us to make black and whitelists we can use in Elastic detection rules. For example a list of developers, compromised accounts, vpn users, etc. We also really like the ability to add metadata to the values in the list.

But atm the Kibana List API is only decently usable with a tool such as Insomnia or Postman, as the Kibana API is not accessible through Dev Tools and there is no gui to manage the Kibana Lists.

Some questions:

• Are there any plans to make gui to manage Kibana Lists?

• If I remember correctly there already is a feature request to make the Kibana API accessible through Dev Tools. Wondering what the status is.. (anyone finds the related GH issue?)

• Are there any plans to merge the Kibana Lists and the ML Filter Lists, which are actually very similar (although the latter seems to lack API?)?

• As we added metadata to some of our list items, we need these fields to be mapped correctly. I'm not sure how do this, considering these are 'system indices" starting with a '.'? For example:

An item we are adding:

{
  "list_id": "list-user-name-compromised-2021",
  "value": "user.name",
	"meta": {
      "comment": "Nigeria",
      "heat_incident_number": "22771",
      "heat_incident_source": "Avanan",
      "user_email": "User.Name@domain.com",
      "user_name_format": "firstname.lastname"
	}
}

But Searching for the list items in discover show the meta fields as unmapped:

Any supported way to make sure these get the correct mapping? What template would we need to edit and how do I apply this template on the .items-default-* indices?

Thanks!

Willem

[ying.mao (Ying M)]

Hi @willemdh!

You can follow the issue for adding support for the Kibana API via Dev Tools here: Console Kibana APIs Support · Issue #49330 · elastic/kibana · GitHub.

I have pinged the team in charge of the Lists API re your other questions and we can hopefully get those answered as well!

Thanks!

[Frank_Hassanabad]

Hi @willemdh ,

I can answer the searchability part of your question about meta. Really interesting how you're using it.

At the moment it is mapped like the following:

From dev tools:

GET .items-default/_mapping

Reading the results, right now we have it as a type "object" but not enabled:

        "meta" : {
          "type" : "object",
          "enabled" : false
        },

However, if you don't have an incredibly large amount of items and just want to search against it you can from DEV tools add a runtime mapping:

Discover | Kibana Guide [7.14] | Elastic

Like so:

However, what that does is add the runtime mapping field to the Kibana Index Pattern (KIP) but not directly on the index its self. So that is a fast convenient way to get your mappings for use with Discover and custom dashboards. Also, if we update/upgrade the mappings you will not have to worry as much about a clash since it stores it in the Kibana index.

However, accessing the KIP (Kibana Index Patterns) runtime fields is not fully supported within the security application just yet. If you want to add them directly to the mapping on the backend you can through here:

# Runtime mapping change to the index its self.
PUT .items-default/_mapping
{
  "runtime": {
    "meta.comment": {
      "type": "keyword"
    },
    "meta.heat_incident_number": {
      "type": "keyword"
    },
    "meta.user_email": {
      "type": "keyword"
    },
    "meta.user_name_format": {
      "type": "keyword"
    }
  }
}

# You can query them now.
GET .items-default/_search
{
  "query": {
    "term": {
      "meta.comment": "<some comment>"
    }
  }
}

Runtime mappings directly on the index will work with any Kibana index pattern and all of the security application.

Outside of this is the classic re-index technique if you have performance issues with runtime mappings/fields. But if you can use runtime fields above and you are ok with your use cases, then I would recommend keepings them As performance becomes a problem, you can then re-index and re-map your index but that leads to higher risk if we change/migrate move these indexes and data into the future.