Kibana dashboard filter doesn't work

tonyaw · April 24, 2023, 2:59pm

I created a Kibana dashboard contains a Lens visualization.
I'm trying to use filter to get data for "cluster_id == 77" OR "cluster_id==80",
But what Lens shows is cluster_id == from 77 to 80. Could you please help to check why?
The inspect request is:

{
  "aggs": {
    "0": {
      "date_histogram": {
        "field": "time",
        "fixed_interval": "12h",
        "time_zone": "UTC",
        "extended_bounds": {
          "min": 1677369609246,
          "max": 1680601527582
        }
      },
      "aggs": {
        "1": {
          "histogram": {
            "field": "cluster_id",
            "interval": 1,
            "min_doc_count": 0,
            "extended_bounds": {
              "min": 77,
              "max": 80
            }
          },
          "aggs": {
            "2": {
              "value_count": {
                "field": "cluster_id"
              }
            }
          }
        }
      }
    }
  },
  "size": 0,
  "fields": [
    {
      "field": "time",
      "format": "date_time"
    }
  ],
  "script_fields": {},
  "stored_fields": [
    "*"
  ],
  "runtime_mappings": {},
  "_source": {
    "excludes": []
  },
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "bool": {
            "should": [
              {
                "match_phrase": {
                  "cluster_id": "77"
                }
              },
              {
                "match_phrase": {
                  "cluster_id": "80"
                }
              }
            ],
            "minimum_should_match": 1
          }
        },
        {
          "range": {
            "time": {
              "format": "strict_date_optional_time",
              "gte": "2023-02-26T00:00:09.246Z",
              "lte": "2023-04-04T09:45:27.582Z"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

stephenb · April 24, 2023, 5:10pm

Hi @tonyaw Welcome to the community.

Did you try to add a filter with the operator is one of

tonyaw · April 25, 2023, 3:03am

Yes, but you can see the filtered result is different at right(from 77 to 80)

stephenb · April 25, 2023, 5:14am

Unfortunately I can not see how you set up the visualization ...

Can you show all the settings/ setup etc

Sample data and mappings

tonyaw · April 25, 2023, 7:12am

Sorry. I may not be able to provide all data for you as it is a product.

My Lens setting:
the response data as Kibana inspect result. Please find related request from my previous comment.

{
  "id": "XXX",
  "rawResponse": {
    "took": 3,
    "timed_out": false,
    "_shards": {
      "total": 1,
      "successful": 1,
      "skipped": 0,
      "failed": 0
    },
    "hits": {
      "total": 64,
      "max_score": null,
      "hits": []
    },
    "aggregations": {
      "0": {
        "buckets": [
          {
            "1": {
              "buckets": [
                {
                  "2": {
                    "value": 0
                  },
                  "key": 77,
                  "doc_count": 0
                },
                {
                  "2": {
                    "value": 0
                  },
                  "key": 78,
                  "doc_count": 0
                },
                {
                  "2": {
                    "value": 0
                  },
                  "key": 79,
                  "doc_count": 0
                },
                {
                  "2": {
                    "value": 0
                  },
                  "key": 80,
                  "doc_count": 0
                }
              ]
            },
            "key_as_string": "2023-02-26T00:00:00.000Z",
            "key": 1677369600000,
            "doc_count": 0
          },

mapping for cluster_id:

{
  "mappings": {
    "properties": {
      "cluster_id": {
        "type": "long"
      },

Don't know if it is enough. Could you please also give me some clue/direction that I can check?
Thanks in advance!

stephenb · April 25, 2023, 11:41am

Hmmm very interesting.

See on the filter where it says Edit in Query DSL can you click on that and share what is in there?

tonyaw · April 25, 2023, 2:25pm

{
  "query": {
    "bool": {
      "should": [
        {
          "match_phrase": {
            "cluster_id": "80"
          }
        },
        {
          "match_phrase": {
            "cluster_id": "77"
          }
        }
      ],
      "minimum_should_match": 1
    }
  }
}

stephenb · April 25, 2023, 3:04pm

What Version of Elastic Stack are you on?

I am on 8.7 and Appears to be working correctly.

I also notice in your picture even though the legend has 4 values the actual graph only has the correct 2... perhaps a Viz bug.

Mine in 8.7 looks correct.... what version are you on?
Did you set any other settings?

I also notice in mine that the response has all the values as well so the filter must be applied after the response comes back which is a bit odd to me.

Request

{
  "aggs": {
    "0": {
      "terms": {
        "field": "http.request.body.bytes",
        "order": {
          "2": "desc"
        },
        "size": 5,
        "shard_size": 25
      },
      "aggs": {
        "1": {
          "date_histogram": {
            "field": "@timestamp",
            "fixed_interval": "30s",
            "time_zone": "America/Los_Angeles",
            "extended_bounds": {
              "min": 1682433656621,
              "max": 1682434556621
            }
          },
          "aggs": {
            "2": {
              "value_count": {
                "field": "http.request.body.bytes"
              }
            }
          }
        },
        "2": {
          "value_count": {
            "field": "http.request.body.bytes"
          }
        }
      }
    }
  },
  "size": 0,
  "fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    },
    {
      "field": "event.created",
      "format": "date_time"
    },
    {
      "field": "event.ingested",
      "format": "date_time"
    },
    {
      "field": "file.accessed",
      "format": "date_time"
    },
    {
      "field": "file.created",
      "format": "date_time"
    },
    {
      "field": "file.ctime",
      "format": "date_time"
    },
    {
      "field": "file.mtime",
      "format": "date_time"
    },
    {
      "field": "process.end",
      "format": "date_time"
    },
    {
      "field": "process.entry_leader.parent.session_leader.start",
      "format": "date_time"
    },
    {
      "field": "process.entry_leader.parent.start",
      "format": "date_time"
    },
    {
      "field": "process.entry_leader.start",
      "format": "date_time"
    },
    {
      "field": "process.group_leader.start",
      "format": "date_time"
    },
    {
      "field": "process.parent.group_leader.start",
      "format": "date_time"
    },
    {
      "field": "process.parent.start",
      "format": "date_time"
    },
    {
      "field": "process.session_leader.parent.session_leader.start",
      "format": "date_time"
    },
    {
      "field": "process.session_leader.parent.start",
      "format": "date_time"
    },
    {
      "field": "process.session_leader.start",
      "format": "date_time"
    },
    {
      "field": "process.start",
      "format": "date_time"
    }
  ],
  "script_fields": {},
  "stored_fields": [
    "*"
  ],
  "runtime_mappings": {},
  "_source": {
    "excludes": []
  },
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "bool": {
            "minimum_should_match": 1,
            "should": [
              {
                "match_phrase": {
                  "http.request.body.bytes": "5"
                }
              },
              {
                "match_phrase": {
                  "http.request.body.bytes": "18"
                }
              }
            ]
          }
        },
        {
          "range": {
            "@timestamp": {
              "format": "strict_date_optional_time",
              "gte": "2023-04-25T14:40:56.621Z",
              "lte": "2023-04-25T14:55:56.621Z"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

Reponse

{
  "id": "FkI2cUJVX09sVFppMTFrcVhYT0k0akEgRm4zRTYzRnhURkNEVG0xWHRSYjlYZzoxNTg4Njk5Njk=",
  "rawResponse": {
    "took": 19,
    "timed_out": false,
    "_shards": {
      "total": 51,
      "successful": 51,
      "skipped": 45,
      "failed": 0
    },
    "hits": {
      "total": 23366,
      "max_score": null,
      "hits": []
    },
    "aggregations": {
      "0": {
        "doc_count_error_upper_bound": 0,
        "sum_other_doc_count": 0,
        "buckets": [
          {
            "1": {
              "buckets": [
                {
                  "2": {
                    "value": 118
                  },
                  "key_as_string": "2023-04-25T07:40:30.000-07:00",
                  "key": 1682433630000,
                  "doc_count": 118
                },
                {
                  "2": {
                    "value": 698
                  },
                  "key_as_string": "2023-04-25T07:41:00.000-07:00",
                  "key": 1682433660000,
                  "doc_count": 698
                },
                {
                  "2": {
                    "value": 626
                  },
                  "key_as_string": "2023-04-25T07:41:30.000-07:00",
                  "key": 1682433690000,
                  "doc_count": 626
                },
                {
                  "2": {
                    "value": 620
                  },
                  "key_as_string": "2023-04-25T07:42:00.000-07:00",
                  "key": 1682433720000,
                  "doc_count": 620
                },
                {
                  "2": {
                    "value": 556
                  },
                  "key_as_string": "2023-04-25T07:42:30.000-07:00",
                  "key": 1682433750000,
                  "doc_count": 556
                },
                {
                  "2": {
                    "value": 480
                  },
                  "key_as_string": "2023-04-25T07:43:00.000-07:00",
                  "key": 1682433780000,
                  "doc_count": 480
                },
                {
                  "2": {
                    "value": 546
                  },
                  "key_as_string": "2023-04-25T07:43:30.000-07:00",
                  "key": 1682433810000,
                  "doc_count": 546
                },
                {
                  "2": {
                    "value": 616
                  },
                  "key_as_string": "2023-04-25T07:44:00.000-07:00",
                  "key": 1682433840000,
                  "doc_count": 616
                },
                {
                  "2": {
                    "value": 494
                  },
                  "key_as_string": "2023-04-25T07:44:30.000-07:00",
                  "key": 1682433870000,
                  "doc_count": 494
                },
                {
                  "2": {
                    "value": 532
                  },
                  "key_as_string": "2023-04-25T07:45:00.000-07:00",
                  "key": 1682433900000,
                  "doc_count": 532
                },
                {
                  "2": {
                    "value": 566
                  },
                  "key_as_string": "2023-04-25T07:45:30.000-07:00",
                  "key": 1682433930000,
                  "doc_count": 566
                },
                {
                  "2": {
                    "value": 580
                  },
                  "key_as_string": "2023-04-25T07:46:00.000-07:00",
                  "key": 1682433960000,
                  "doc_count": 580
                },
                {
                  "2": {
                    "value": 512
                  },
                  "key_as_string": "2023-04-25T07:46:30.000-07:00",
                  "key": 1682433990000,
                  "doc_count": 512
                },
                {
                  "2": {
                    "value": 592
                  },
                  "key_as_string": "2023-04-25T07:47:00.000-07:00",
                  "key": 1682434020000,
                  "doc_count": 592
                },
                {
                  "2": {
                    "value": 464
                  },
                  "key_as_string": "2023-04-25T07:47:30.000-07:00",
                  "key": 1682434050000,
                  "doc_count": 464
                },
                {
                  "2": {
                    "value": 486
                  },
                  "key_as_string": "2023-04-25T07:48:00.000-07:00",
                  "key": 1682434080000,
                  "doc_count": 486
                },
                {
                  "2": {
                    "value": 646
                  },
                  "key_as_string": "2023-04-25T07:48:30.000-07:00",
                  "key": 1682434110000,
                  "doc_count": 646
                },
                {
                  "2": {
                    "value": 548
                  },
                  "key_as_string": "2023-04-25T07:49:00.000-07:00",
                  "key": 1682434140000,
                  "doc_count": 548
                },
                {
                  "2": {
                    "value": 614
                  },
                  "key_as_string": "2023-04-25T07:49:30.000-07:00",
                  "key": 1682434170000,
                  "doc_count": 614
                },
                {
                  "2": {
                    "value": 556
                  },
                  "key_as_string": "2023-04-25T07:50:00.000-07:00",
                  "key": 1682434200000,
                  "doc_count": 556
                },
                {
                  "2": {
                    "value": 514
                  },
                  "key_as_string": "2023-04-25T07:50:30.000-07:00",
                  "key": 1682434230000,
                  "doc_count": 514
                },
                {
                  "2": {
                    "value": 524
                  },
                  "key_as_string": "2023-04-25T07:51:00.000-07:00",
                  "key": 1682434260000,
                  "doc_count": 524
                },
                {
                  "2": {
                    "value": 636
                  },
                  "key_as_string": "2023-04-25T07:51:30.000-07:00",
                  "key": 1682434290000,
                  "doc_count": 636
                },
                {
                  "2": {
                    "value": 720
                  },
                  "key_as_string": "2023-04-25T07:52:00.000-07:00",
                  "key": 1682434320000,
                  "doc_count": 720
                },
                {
                  "2": {
                    "value": 584
                  },
                  "key_as_string": "2023-04-25T07:52:30.000-07:00",
                  "key": 1682434350000,
                  "doc_count": 584
                },
                {
                  "2": {
                    "value": 624
                  },
                  "key_as_string": "2023-04-25T07:53:00.000-07:00",
                  "key": 1682434380000,
                  "doc_count": 624
                },
                {
                  "2": {
                    "value": 523
                  },
                  "key_as_string": "2023-04-25T07:53:30.000-07:00",
                  "key": 1682434410000,
                  "doc_count": 523
                },
                {
                  "2": {
                    "value": 568
                  },
                  "key_as_string": "2023-04-25T07:54:00.000-07:00",
                  "key": 1682434440000,
                  "doc_count": 568
                },
                {
                  "2": {
                    "value": 700
                  },
                  "key_as_string": "2023-04-25T07:54:30.000-07:00",
                  "key": 1682434470000,
                  "doc_count": 700
                },
                {
                  "2": {
                    "value": 638
                  },
                  "key_as_string": "2023-04-25T07:55:00.000-07:00",
                  "key": 1682434500000,
                  "doc_count": 638
                },
                {
                  "2": {
                    "value": 714
                  },
                  "key_as_string": "2023-04-25T07:55:30.000-07:00",
                  "key": 1682434530000,
                  "doc_count": 714
                }
              ]
            },
            "2": {
              "value": 17595
            },
            "key": 18,
            "doc_count": 17595
          },
          {
            "1": {
              "buckets": [
                {
                  "2": {
                    "value": 12
                  },
                  "key_as_string": "2023-04-25T07:40:30.000-07:00",
                  "key": 1682433630000,
                  "doc_count": 12
                },
                {
                  "2": {
                    "value": 186
                  },
                  "key_as_string": "2023-04-25T07:41:00.000-07:00",
                  "key": 1682433660000,
                  "doc_count": 186
                },
                {
                  "2": {
                    "value": 198
                  },
                  "key_as_string": "2023-04-25T07:41:30.000-07:00",
                  "key": 1682433690000,
                  "doc_count": 198
                },
                {
                  "2": {
                    "value": 224
                  },
                  "key_as_string": "2023-04-25T07:42:00.000-07:00",
                  "key": 1682433720000,
                  "doc_count": 224
                },
                {
                  "2": {
                    "value": 218
                  },
                  "key_as_string": "2023-04-25T07:42:30.000-07:00",
                  "key": 1682433750000,
                  "doc_count": 218
                },
                {
                  "2": {
                    "value": 194
                  },
                  "key_as_string": "2023-04-25T07:43:00.000-07:00",
                  "key": 1682433780000,
                  "doc_count": 194
                },
                {
                  "2": {
                    "value": 192
                  },
                  "key_as_string": "2023-04-25T07:43:30.000-07:00",
                  "key": 1682433810000,
                  "doc_count": 192
                },
                {
                  "2": {
                    "value": 226
                  },
                  "key_as_string": "2023-04-25T07:44:00.000-07:00",
                  "key": 1682433840000,
                  "doc_count": 226
                },
                {
                  "2": {
                    "value": 174
                  },
                  "key_as_string": "2023-04-25T07:44:30.000-07:00",
                  "key": 1682433870000,
                  "doc_count": 174
                },
                {
                  "2": {
                    "value": 172
                  },
                  "key_as_string": "2023-04-25T07:45:00.000-07:00",
                  "key": 1682433900000,
                  "doc_count": 172
                },
                {
                  "2": {
                    "value": 220
                  },
                  "key_as_string": "2023-04-25T07:45:30.000-07:00",
                  "key": 1682433930000,
                  "doc_count": 220
                },
                {
                  "2": {
                    "value": 208
                  },
                  "key_as_string": "2023-04-25T07:46:00.000-07:00",
                  "key": 1682433960000,
                  "doc_count": 208
                },
                {
                  "2": {
                    "value": 228
                  },
                  "key_as_string": "2023-04-25T07:46:30.000-07:00",
                  "key": 1682433990000,
                  "doc_count": 228
                },
                {
                  "2": {
                    "value": 212
                  },
                  "key_as_string": "2023-04-25T07:47:00.000-07:00",
                  "key": 1682434020000,
                  "doc_count": 212
                },
                {
                  "2": {
                    "value": 188
                  },
                  "key_as_string": "2023-04-25T07:47:30.000-07:00",
                  "key": 1682434050000,
                  "doc_count": 188
                },
                {
                  "2": {
                    "value": 192
                  },
                  "key_as_string": "2023-04-25T07:48:00.000-07:00",
                  "key": 1682434080000,
                  "doc_count": 192
                },
                {
                  "2": {
                    "value": 168
                  },
                  "key_as_string": "2023-04-25T07:48:30.000-07:00",
                  "key": 1682434110000,
                  "doc_count": 168
                },
                {
                  "2": {
                    "value": 156
                  },
                  "key_as_string": "2023-04-25T07:49:00.000-07:00",
                  "key": 1682434140000,
                  "doc_count": 156
                },
                {
                  "2": {
                    "value": 144
                  },
                  "key_as_string": "2023-04-25T07:49:30.000-07:00",
                  "key": 1682434170000,
                  "doc_count": 144
                },
                {
                  "2": {
                    "value": 150
                  },
                  "key_as_string": "2023-04-25T07:50:00.000-07:00",
                  "key": 1682434200000,
                  "doc_count": 150
                },
                {
                  "2": {
                    "value": 192
                  },
                  "key_as_string": "2023-04-25T07:50:30.000-07:00",
                  "key": 1682434230000,
                  "doc_count": 192
                },
                {
                  "2": {
                    "value": 180
                  },
                  "key_as_string": "2023-04-25T07:51:00.000-07:00",
                  "key": 1682434260000,
                  "doc_count": 180
                },
                {
                  "2": {
                    "value": 192
                  },
                  "key_as_string": "2023-04-25T07:51:30.000-07:00",
                  "key": 1682434290000,
                  "doc_count": 192
                },
                {
                  "2": {
                    "value": 208
                  },
                  "key_as_string": "2023-04-25T07:52:00.000-07:00",
                  "key": 1682434320000,
                  "doc_count": 208
                },
                {
                  "2": {
                    "value": 206
                  },
                  "key_as_string": "2023-04-25T07:52:30.000-07:00",
                  "key": 1682434350000,
                  "doc_count": 206
                },
                {
                  "2": {
                    "value": 240
                  },
                  "key_as_string": "2023-04-25T07:53:00.000-07:00",
                  "key": 1682434380000,
                  "doc_count": 240
                },
                {
                  "2": {
                    "value": 161
                  },
                  "key_as_string": "2023-04-25T07:53:30.000-07:00",
                  "key": 1682434410000,
                  "doc_count": 161
                },
                {
                  "2": {
                    "value": 172
                  },
                  "key_as_string": "2023-04-25T07:54:00.000-07:00",
                  "key": 1682434440000,
                  "doc_count": 172
                },
                {
                  "2": {
                    "value": 172
                  },
                  "key_as_string": "2023-04-25T07:54:30.000-07:00",
                  "key": 1682434470000,
                  "doc_count": 172
                },
                {
                  "2": {
                    "value": 186
                  },
                  "key_as_string": "2023-04-25T07:55:00.000-07:00",
                  "key": 1682434500000,
                  "doc_count": 186
                },
                {
                  "2": {
                    "value": 200
                  },
                  "key_as_string": "2023-04-25T07:55:30.000-07:00",
                  "key": 1682434530000,
                  "doc_count": 200
                }
              ]
            },
            "2": {
              "value": 5771
            },
            "key": 5,
            "doc_count": 5771
          }
        ]
      }
    }
  },
  "isPartial": false,
  "isRunning": false,
  "total": 51,
  "loaded": 51,
  "isRestored": false
}

tonyaw · April 26, 2023, 2:57am

Kibana version I'm using is v 8.6.0. Could you please help to confirm if it is a 8.6 bug or not?
Elasticsearch/Kibana upgrade always not easy.
Please refer to my previous update:
Kibana dashboard filter doesn't work - #5 by tonyaw
The result returned from Elasticsearch contains data from 77 to 80.
So if it is a bug, it shall be ES bug instead of Kibana bug.
What do you think?

stephenb · April 26, 2023, 4:21am

Here is 8.6.0

It looks like it works as it should to me... I think you have something else ... some other setting...

One thing is odd you are treating a long like a keyword... typically a container.id would be of type keyword and that would be used to filter... you are treating a number like a keyword but is still does work.

So I used the Sample Web Logs under Try Sample Data from the Kibana Home Scree.

a data set that is included you could try.

I think you have some other setting or something.... I do not think there is a bug.

Here is the basic equivalent,
Date Histogram
Vertical is Count of Bytes (You should really just do Count of Records)
Then breakdown by Top 5 Byte (which is an number like your container id)
Then the filter and it works..

Without filter...

With Filter

tonyaw · April 26, 2023, 5:44am

Thanks for your reminder.
Following is the root cause. I used "Intervals" instead of "Top values" wrongly.
After change it back to "Top values", Filter works fine. Thanks for your great support!
kibana_issue

system · May 24, 2023, 5:44am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Kibana filters in Dashboard vs Lens (aggs vs query) Kibana lens , dashboard	2	388	March 28, 2023
Kibana Dashboard shows no result even if there is non-zero number of hits Kibana lens , kql-kibana-query-language , dashboard	3	702	April 10, 2023
Kibana visualization with "filter by condition" doesn't work on dashboard Kibana	3	846	January 21, 2020
How to filter by "Count of records" within Kibana Lens Kibana lens	4	2933	July 25, 2023
Filter working in "Discover" field, but the same filter in Dashboard/lense does not Kibana	8	322	March 17, 2023

Kibana dashboard filter doesn't work

Related topics