How to query all unique IP addresses

[jeromeat]

I'm looking for help on how to query all unique IP addresses, however, the field types that the IPs get parsed to are setup as "keyword" and not "IP". And they are setup as ip_src, ip_dst, dst_ip, src_ip.

Yes, I understand this probably isn't ideal, but this can't be changed and out of my control, so I'm hoping there is still a way to dump all the IPs seen in those fields. Also, can this be queried via a CIDR range. I'm thinking not since the fields are not set to "IP", but thought I'd ask.

Thanks in advance.

[stephenb (Stephen Brown)]

@jeromeat

Hi there ... I love these question because to me they lead to many more...

I will answer the first and easiest and show you cardinality and top hits (think unique) IPs.

(And yes converting these fields to actual ip data type would provide much more powerful searching and grouping but we can talk about that later, perhaps even how to do it after the data is ingested)

But when I read ALL, is all 1, 1000, 1,000,000 IPs do you really want to dump and display 1M IPs? Over what time frame? (BTW there are some limitation at extreme scale with aggregations which is what we are talking about be we will talk about that later if it comes up.

Typically folks are look at Top 100 Busy, Bottom 100 etc.. etc..

Also you can do this very easily in Lens Visualization it is built to answer these types of questions... You can build Tables or Bar charts etc... For almost every network team builds these kind of visualizations and table.

But here is a simple example of what you asked.

DELETE discuss

PUT discuss
{
  "mappings": {
    "properties": {
      "src_ip": {
        "type": "keyword"
      }
    }
  }
}

POST discuss/_doc
{
  "src_ip" : "10.0.0.1"
}

POST discuss/_doc
{
  "src_ip" : "10.0.0.2"
}

POST discuss/_doc
{
  "src_ip" : "10.0.0.1"
}

POST discuss/_doc
{
  "src_ip" : "10.0.0.3"
}

POST discuss/_doc
{
  "src_ip" : "10.0.0.2"
}

# Cardinality
GET discuss/_search
{
  "size": 0,
  "aggs": {
    "unique_ips": {
      "cardinality": {
        "field": "src_ip"
      }
    }
  }
}

# Unique IPs / Top Hits
POST /discuss/_search
{
  "size": 0,
  "aggs": {
    "top_tags": {
      "terms": {
        "field": "src_ip"
      },
      "aggs": {
        "top_ips": {
          "top_hits": {
            "size": 1
          }
        }
      }
    }
  }
}

[jeromeat]

@stephenb Thanks for the quick reply. I'm actually looking to list out all the unique IPs. I have been able to use cardinality to get totals.

This query is just for demonstration purposes on lab equipment and wouldn't be used for thousands or more IPs.

Also, this query is going to be called from another non-kibana dashboard probably using curl to get this information. I did see that there is a "multi_terms" agg, but doesn't seem to be in version 7.11, so I got excited, then sad.

[stephenb (Stephen Brown)]

Below will give the actual IPs, you may need to page through them but I think you can use this up to 10,000 see below I set it too1000 that would give you up to 1000 unique IPs.

not sure what you want to do with multi_term ....

@stephenb

# Unique IPs / Top Hits
POST /discuss/_search
{
  "size": 0,
  "aggs": {
    "top_tags": {
      "terms": {
        "field": "src_ip"
      },
      "aggs": {
        "top_ips": {
          "top_hits": {
            "size": 1000
          }
        }
      }
    }
  }
}

If you were on a newer version like 7.14.+ here is how to a runtime mapping and turn it into an actual ip data type but in 7.11 you would need to use and ingest pipeline and add the new fields.

PUT discuss/_mapping
{
  "runtime": {
    "src_ip_fixed": {
      "type": "ip",
      "script": {
        "source": "emit(doc['src_ip'].value)"
      }
    }
  }
}

GET discuss/_search
{
  "query": {
    "range": {
      "src_ip_fixed": {
        "gte": "10.0.0.0",
        "lte": "10.0.0.2"
      }
    }
  }
}

GET discuss/_search
{
  "query": {
    "term": {
      "src_ip_fixed": "10.0.0.0/16"
    }
  }
}

[leandrojmp (Leandro Pereira)]

Which version are you using? From 7.12+ you can use runtime fields to map those fields as IP and query using CIDR for example.

You would need to do something like this:

PUT index-name/_mapping
{
  "runtime": {
    "field.name": {
    "type": "ip"
    }
  }
}

[stephenb (Stephen Brown)]

@leandrojmp

Which version are you using? From 7.12+ you can use runtime fields to map those fields as IP and query using CIDR for example.

Ha thanks @leandrojmp I was thinking 7.14. thanks for reminding me.

I gave @jeromeat and exact example for runtime above... you still need to emit the value etc... and I showed even how to do searches!

PUT discuss/_mapping
{
  "runtime": {
    "src_ip_fixed": {
      "type": "ip",
      "script": {
        "source": "emit(doc['src_ip'].value)"
      }
    }
  }
}

[jeromeat]

Thanks all. Unfortunately we are at 7.11. Hopefully we will upgrade soon.