How to check if a field value of a tag is present in the field value of other tags

#1

Hi,

I want to identify if an IP address(field A) is present in the list of IP addresses(field B).
How will I be able to do this check?

  1. How should I convert the IP addresses present in a field B into a list first?
  2. Once the list is identified, how will I be able to compare field A ip to the field B ipaddress list?

Need your help on the same.

Thanks in advance!

#2

Hey,

there are basically two solutions to this: at index time and at query time, if your document contains the list of ip adresses as well as the ip address to check for. Take this example

DELETE test

PUT test/_doc/1
{
  "address" : "1.2.3.4",
  "addresses" : [ "1.2.3.4", "8.8.8.8", "4.4.4.4" ]
}

PUT test/_doc/2?refresh=true
{
  "address" : "4.3.2.1",
  "addresses" : [ "1.2.3.4", "8.8.8.8", "4.4.4.4" ]
}

GET test/_search
{
  "query": {
    "bool": {
      "filter": {
        "script": {
          "script": {
            "lang": "painless",
            "source": "return doc['addresses.keyword'].value.contains(doc['address.keyword'].value)"
          }
        }
      }
    }
  }
}

this uses a script filter to check if an IP is with in the list. While this works, it is a bit slower, because you basically have to execute the script for each document.

An alternative is to store this information at index time using an ingest pipeline

DELETE test2

PUT _ingest/pipeline/ip_pipeline
{
  "processors": [
    {
      "script": {
        "lang": "painless",
        "source": "ctx.contains_ip_address = ctx.addresses.contains(ctx.address)"
      }
    }
  ]
}

PUT test2/_doc/1?pipeline=ip_pipeline
{
  "address" : "1.2.3.4",
  "addresses" : [ "1.2.3.4", "8.8.8.8", "4.4.4.4" ]
}

PUT test2/_doc/2?pipeline=ip_pipeline&refresh=true
{
  "address" : "4.3.2.1",
  "addresses" : [ "1.2.3.4", "8.8.8.8", "4.4.4.4" ]
}

GET test2/_search
{
  "query": {
    "bool": {
      "filter": {
        "term": {
          "contains_ip_address": "true"
        }
      }
    }
  }
}

the query in this case will be much faster.

hope this helps.

#3

Alexander - Thank you so much! But how to make the ips in a field into a list?

I have a filebeat pushing logs into logstash inturn to elasticsearch with a field in which ips are stored(i.e. ipaddr:1.2.3.4). How will I be able to convert all the individual ips in the field name ipaddr to a list?

#4

I guess I misunderstood your use case. Are you just trying to find documents that contain a certain IP then? Can you share a sample document and a sample query?

#5

My usecase here is -

I need to write a watcher such that an alert should be triggered whenever an IP(field A), is present in list of IPs(field B).

Eg - If 1.2.3.4 is present in [1.2.3.4, 2.4.5.6, 3.7.8.9] then the condition should meet and alert will be triggered.

In here, the confusion is

  1. I have an index(index:sample) and field ipaddr which contains ipaddress(ipaddr:1.2.3.4), the index has finite number of events(say 200 ipaddress). Now the confusion is how will I be able to convert all the values of ipaddr field into a list.?

  2. Once if that list is obtained, I have to check if an IP(newly indexed in an event) is present in that list(list of IPs in field B).

Hope this clarifies

#6

Hey,

ah, so the list of ip addresses is dynamic, not static within the document. Take a look at the terms query. In order to construct that filter from this index you can query that index first, and then use a transforming the chained input data, to construct the proper list for the terms filter. Note: I have not tested this, but this is what I would try.

Hope this helps!

--Alex