I want to search the index for the same value as the document in another index

Elastic Stack Elasticsearch
1

Hello.
I put a proxy log in the elasticsearch index.
Next, I made an index with blacklist addresses.
Is it possible to search addresses in the blacklist index from the proxy log index?

Thank you.
kobaya

2

If your blacklist is small then it is most efficient to write a loop in your client that reads the addresses and issues searches. Multiple addresses can be grouped using a terms query for more efficient querying.
If your blacklist is large then it may be faster to compare two sorted streams of information rather than doing lots of searches (random disk seeks are expensive). This can be done using something like the composite aggregation with the after parameter:

GET testproxy,testblacklist/_search
{
  "size":0,
  "aggs": {
	"joined": {
	  "composite": {
		"sources": [
		  {
			"address": {
			  "terms": {
				"field": "address",
				"order": "asc"
			  }
			}
		  }
		]
	  },
	  "aggs": {
		"numIndices": {
		  "cardinality": {
			"field": "_index"
		  }
		}
	  }
	}
  }
}

The disadvantage with composite is that you can't trim the JSON of addresses that only got a hit on one of the indices.
Alternatively, you could use the terms aggregation with partitioning and sort the addresses by the number of matched indices to help trim the non-interesting addresses from the results. This wizard might help with some of the join choices.

1 Like
3

Hi Mark_Harwood,

Thanks for the reply.
I would like to try what you taught me.

Thank you

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.