Creating a filter based on two indexes

Elastic Stack Kibana
1

Hello,

we have 2 indexes and we want to create a filter that will compare and match specific fields from the first index with the second index and the created filter should be boolean (yes and no).

the created filter should exist in the first index which is the data (the second index is a CSV file)

any help on how to do that?

2

You can elaborate on your specific use case? A filter is always executed on a per-document level defining static criteria for the fields of the document, so I'm not sure what you mean by comparing fields to another index.

3

so we have two indexes

the first index is our data which has some traffic on it
the second index is our csv file that we have ingested through logstash by @ timestamp

in our csv index we only have 1 field which is 'domain' we want to create a filter such that it will take this field and compare against other fields in our traffic

it's should be visible like any other filter we see in kibana.

4

I see, makes sense. Unfortunately filters can't do this - they are just static criteria descriptors and are not able to look up dynamic data.

As you probably know, you can add filter pills to Kibana dashboards and save the dashboard along with the filter. The next user accessing the dashboard will see the data filtered down according to the added filter.

If you want to programmatically change this filter without going through the UI (I'm assuming this is your ultimate goal), you could also use the saved objects api to download the dashboard json, change the filter, and upload it again - the script doing this would replace your logstash instance ingesting the csv file.

Does this cover your use case?

5

Great !

but my concern is is it possible for us to create this based on two indexes ?

also i came across logstash translate filter plugin and it has some of the feature that we want but am not really sure how to implement it between two indexes and what is the end result ...

the two indexes situation really confuses me.

6

[...] create this based on two indexes

As you would have to write the script calling the saved objects api and adding the filter yourself in your programming language of choice, you can do whatever you please, of course that includes looking up the value from another Elasticsearch index (by sending a search request to Elasticsearch and using the response) or by reading it from a local csv file.

But when uploading the dashboard again it has to contain the current filter value as static json (without any dynamic lookup - that's just not possible).

The logstash translate filter plugin will just do a static lookup of a value at ingest time - this means the current value will be stored in every single document in Elasticsearch and you can't simply change it later without updating every single document.

I think the confusion here comes from the fact that you basically want to do a join here - looking up values from another index while fetching data from an index. This is not possible in Elasticsearch by design - as Elasticsearch is a distributed database, it's inefficient to access values from other documents as they might be stored on a whole different machine.

7

Thank you so much you have been a great help!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.