How to remove values from a returned search if they are equal?

[mds4872 (Mark)]

Hi yall,

I have traceroute data pointing to websites and I wanted to know how I could filter out responses if they are qual in hop length? For example, I have a job-id with attached http and https traceroutes, once each traceroute completes it returns a total-hops value. I want to filter out all job-id's where the total-hops value for https and http are equal. Below is an example of what the table format looks like for an https result. The Job-id remains the same for http. Basically for each job-id I want to find results where total-hops for http != https.

I also built a data table to also show what im looking to do. In this image, it is showing a count of total hops broken down by job-id and protocol.keyword. You can see that http and https hops are equal so I would like to exclude them from the visualization. I've tried messing around with the request field itself, but I can't seem to get it to exclude the job-id if the hop counts are equal. Any help or ideas would be appreciated!

[tsullivan (Tim Sullivan)]

I don't think Kibana table cells are configurable in the way you're thinking. There are ways to craft requests to Elasticsearch that can apply scripts in sub-aggregations and calculate over the values of higher-level aggregations. Those scripts generally work by taking metric inputs and returning a metric output. One example is the Bucket Script aggregation. Various visualization types in Kibana are integrated with that aggregation type, and playing around with those aggregation types could help you move forward.

Another idea is to try a scripted field, or have a pre-processing step at data ingestion time that can calculate a new field out of the raw data that gives better semantic value. Right now, it seems like you're not sure when http / https counts are relevant for the visualization, but maybe you need a field that only exists when those counts are relevant, and can be used in the visualization in place of http / https.

Sorry, I'm not sure if these ideas apply directly to your use case, but maybe it can help rethink the problem in a way that's more natural to Elasticsearch aggregations: calculations have to be factored in at an early step.

[mds4872 (Mark)]

Hi Tim,
Thanks for the tips! We actually did end up with a pre-processing check from our redis db before we shipped the data to ELK. Now the data will have a boolean value that indicates whether there was a differnce in the hop counts or not, and we can just visualize the data using that boolean value. I think we couldve done it with a scripted field, but we were already doing a bunch of other pre-processing with python so it was easiest to just add the logic in there.

Thanks again for the help!