Correlation in ELK dashboard

Elastic Stack Kibana
#1

Hello All,
I have an index with below fields and data set,

    1.indicator:13.12.14.15
	  action:success
	
	2.indicators:13.12.14.15
	  tags:elk-search
	
	3.indicator:14.4.5.67.84
	  action:success

I have built a dashboard for the above with 4 panels indicator,indicators,action and tags.
Now if I search for success only 2 panels are filtering data i.e,

      indicator:13.12.14.15
	  action:success

But I wanted data in 4 panels because success is related to 13.12.14.15 and the IP is present in one more meta called indicators which is related to tags.
To summarize if i filter success in my dashboard I need to get the below correlated data,

    indicator:13.12.14.15
	action:success
	indicators:13.12.14.15
	tags:elk-search

Is there any ways to achieve this using ELK queries? Kindly suggest your ideas as I am a beginner in writing queries.

#2

Hi Sivajanani, it sounds like the data is fragmented and you need a way to connect it back together so it can be searched effectively. Kibana doesn't have a way to connect the fragments for you in its visualizations.

I think your best bet is to re-index the data using a query that groups the data into single entities. If indicator is a numeric type orkeyword type field, and and it is how you want the data to be connected, you could run this query to put the entities together:

POST /data-index-*/_search
{
  "size": 5000,  // search for 5000 indicators
  "query": {
    "match_all": {}
  },
  "collapse": {
    "field": "indicator", // designate indicator as the field to group other data by
    "inner_hits": {
      "name": "most_recent",
      "size": 100, // find up to 100 other documents with the same indicator
      "sort": [ { "@date": "desc" } ] // sort with newest documents first
    }
  },
  "sort": [
    {
      "indicator": {
        "order": "asc"
      }
    }
  ]
}

My advice is to create a script that runs that query, and takes the inner_hit JSON data to create new documents, and save those documents in a new Elasticsearch index.

#3

Hello @tsullivan
Thanks for the solution. But still I will be able to get only the indicators field.
So I tried another way around using join. But still I am facing issues. my join field is created but am not able to correlate still.
I think I have missed out some concepts. Please help.

#4

I encourage you to look more into it. With field collapsing, you will get each entire document that correlates to the same indicator value.

See Expand Collapse Results