Question on filtering the documents in ML Data feed

Elastic Stack Kibana
1

Hi, I have scenario where i want to create the ML job for specific group of servers for metricbeat index.
Let's suppose i have server1 to server7 which are grouped and specified to application1 and from server8 to server15 related to application2.
Now i want to create ML jobs for system cpu/network/disk by grouping severs based on host names. how can we pass the input filter in data feed.

2

In general, if you want to filter, you

  1. create a Saved Search in Kibana Discover with the proper filters - name that saved search. This will obviously create a subset of the entire data.
  2. create the ML job, but instead of selecting the job to use the full index, have it use the named saved search
3

Hi richcollier, the above solution is working quiet good. But what if i have 30 servers to filter. Internally ELK is creating the bool query in data feed while creating ML job. Is there any another way by using the lookups and passing the list of servers in an array.

I have created the lookup and trying to pass the servers in an array. below is the lookup sample i have created and passing these in the data feed.

        lookup:
        POST machine-learning-lookups/_doc/metricbeat_xxx_cpu
        {
          "metricbeat_xxx_tag_names":["server1","server2", ...........]
          
          
        } 

        Datafeed query:
        {
       "query": {
  "bool": {
    "filter": [
      {"term": {  "event.dataset": "system.cpu"}},
      {"terms" : {"type_instance": {
				  "index":"machine-learning-lookups",
				  "type": "_doc",
				  "id": "metricbeat_xxx_cpu",
				  "path": "metricbeat_xxx_tag_names"
				}} }
    ],
    "must": {
      "exists": {"field": "system.cpu.total.norm.pct"}
    }
    
  }
  } 
    }

Results:
{
  "took" : 1312,
  "timed_out" : false,
  "num_reduce_phases" : 2,
  "_shards" : {
    "total" : 956,
    "successful" : 956,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

the above one is not giving any docs in Data preview for ML

4

See the following blog for an example of building a look-up index and using that as a way to filter the datafeed:

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.