Filter out documents in Discover view that match a set of saved queries/searches?

Elastic Stack Kibana
1

My team is running a logging stack, where they need to follow up on error events in Kibana.

Unfortunately, some errors take a while to get fixed, and some actually never get fixed because they are caused by external events. It is possible to filter out documents with a saved search or query, but after a while this saved search becomes too large and unpractical to maintain.

Is it possible to do a search and only return documents that do not match any of a certain set of saved queries or filters? Ideally, We would be creating specific filters for specific error messages, named after a follow-up ticket, so that we can still see how often they happen in a dashboard, but they wouldn't "pollute" the view in Discover.

Or is the only way to do this to build and maintain one single huge query?

2

Some ideas. My colleagues may have more. I recommend solving this at the source where the logging data is coming in. Don't index what you don't need. There are a variety of options depending on what you are using to ingest data

If you don't want to solve this at the source where the data is feeding in, you could also do something more fancy as the data is indexing. check out drop processors as part of indexing pipelines.

Final delete-focused idea is to delete documents by query periodically....this is pretty simple but you have to run it periodically

I'm sure your long KQL filter is well tested but regardless of which one of these you choose make sure your conditions are working EXACTLY the way you expect them to as these methods all will simply not index the data. Any mistakes or over-acting rules would be deleting data.

If you didn't want to solve it either of these ways (you want to keep the documents) my only other idea is a "latest" Transform (see step 8) that also allows you to enter a filter condition and filter out older data. This could keep the index you're transforming data into "clean" while retaining the source data. This is another index, though...so consider that it'll be more space.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.