Searching multiple fields(x,y) in Elasticsearch index "A" for entries in present in a field(z) in different index "B"

Elastic Stack Elasticsearch
1

Hi,

I have a index which captures logs from various security devices at the perimeter. I want to check for possible connections from/to blacklisted IPs.

I have a index, which has the logs of the device and another index which contains blacklisted IPs ( updated daily).

How do I filter device logs for connections to the blacklisted IP addresses.

I'm a newbie to ELK stack. Can you please help me out with this.

Thanks
Natarajan

2

Hello i already had this use case, and for the query i used https://www.elastic.co/guide/en/elasticsearch/reference/1.4/query-dsl-terms-filter.html#_terms_lookup_twitter_example

3

Hi Camilo,

Should the field names in both the indices be same ?
The query I tried based on the term filter is not working as expected. Please let me know if I have made any mistake in the query.

Device logs -
Index name : device_logs
Fields to be filtered : source_address, destination_address

Blacklisted IP -
Index name : blacklisted_ip
Field to be used for filtering - ip_address

Please find the query below -
I have used "id" : "*" for filtering on all the documents in the field ip_address

curl -XGET localhost:9200/device_logs/_search -d '{
{
"query": {
"filtered": {
"filter": {
"terms": {
"logs": {
"index": "blacklisted_ip",
"type": "logs",
"id": "*",
"path": "ip_address"
},
"_cache_key": "blacklst_ip_match"
}
}
}
}
}

4

what i made is in your index blacklisted_ip i keep a inner object with all the blacklist ip_address, and after in id i give the id of my inner object.

i'm not sure that the "*" in the Id works, and if it works not sure that is a good idea.

5

Thanks for the inputs Camilo ... I'm a newbie and I have not worked with inner objects in ES..Is there any ES guide for that where I can familiarise with the concepts... I'm not able to find any examples on inner objects for this use case. Please let me know..

And, I'm currently using logstash for loading data to ES.. Is there any mechanism to index these entries from Logstash to ES index inner objects ??

PS : I'm currently using ES 2.2 where terms filter has been replaced by terms query.

6

hear can help you to understand, https://www.elastic.co/blog/managing-relations-inside-elasticsearch

and for your use case hear using Array (string in the mapping ES):

curl -XPUT localhost:9200/ip_list/blacklisted/1 -d '{
   "ip_address" : ["127.0.0.0", "127.0.0.1"]
}'


curl -XPUT localhost:9200/device_logs/logs -d '{
   "text" : "test test test",
   "ip" : "127.0.0.0"
}'


curl -XGET localhost:9200/device_logs/logs/_search -d '{
  "query" : {
    "filtered" : {
      "filter" : {
        "terms" : {
          "ip" : {
            "index" : "ip_list",
            "type" : "blacklisted",
            "id" : "1",
            "path" : "ip_address"
          }
        }
      }
    }
  }
}'