How to use translate plugin with ES index for lookup data

[msk_76]

I have read about translate filter functionality in logstash which could be used for lookup event data with that of a file( csv, json) which we can define in destination_path yml file.
Can I use an elastisearch index ( containing the key-value lookup data) to be used instead of csv/json file?

This is due to the reason that my event data is in one index and my lookup data is in another index and I want to merge them based on key value. The lookup data is huge to be kept in a csv/json file.

if it's not possible with translate, is there any other way or other filter?

[Christian_Dahlqvist]

You will need to use the Elasticsearch filter plugin. Be aware that this can be slow though as it requires a network round trip per event.

[msk_76]

I read about the elasticsearch plugin filter but it seems that it can refer to old events in same index.
My requirement is to do lookup based on timestamp, cluster,

index1
timestamp|cluster|jobid|usage

index2
timestamp|cluster|jobid|hostname

index3
hostname|hostgroup

Logic of lookup is
if max(timestamp) of index2 > max(timestamp) of index2 then
left outer join index1 with index2 with index3 on
index1.timestamp = index2.timestamp and index1.cluster = index2.cluster and index1.jobid = index2.jobid
index2.hostname = index3.hostname

The resultant output would be index4
timestamp|cluster|jobid|usage|hostname|hostgroup

[Christian_Dahlqvist]

Elasticsearch does not support joins so you may need to express this as multiple separate Elasticsearch lookups. I not sure I follow the logic so may be wrong though.

[msk_76]

Ok I can do it with multiple ES lookups connected sequentially but how to implement this condition

if [max(timestamp)] of index2 > [max(timestamp)] of index1 ?

timestamp is the field present in both index1 & index2. If the above condition is satisfied then only parse the logstash input.

Please suggest.

[Christian_Dahlqvist]

I do not think you can do that so may need to restructure the indexes.