Using bucket aggregation on IPs not giving expected results

Elastic Stack Elasticsearch
1

image
image.png1244×547 28.6 KB

Hello,

I am trying to search for documents containing an IP which falls under a range. As specified in the image the docs have IP 192.168.255.100 which falls under the mask: 192.168.0.0/24. But the result that I get is not as per my understanding. I was expecting to get docs in the bucket. Moreover if I try with 192.168.0.0/24 I get ArrayOutOfBoundIndexException. Not sure what's happening. Also from and to string is weird with some different
encoding.

Could you please point me out as to where its getting wrong?

Thanks

2

The name of the field that you run this aggregation against (ipAddress.keyword) suggests that you are running this aggregation against a field that has been mapped as type keyword. You need to map this field as type ip if you want to use the ip_range aggregation.

3

If I use ip type in aggregation it gives me error saying "Fielddata is disabled on text fields by default. Set fielddata=true on [ipAddress] in order to load fielddata in memory by uninverting the inverted index"

I don't want to use fielddata because it would be costly in terms of memory. So I opted for multi-field and used keyword subfield to use in aggs.

4

I think your field ipAddress is actually mapped as a type text rather than as type ip. You could check that by retrieving the mapping:

GET hostaddressindex/client/_mapping

I think you will see "type": "text" instead of "type": "ip" for the ipAddress field.

You will have to change your mapping such that ipAddress is mapped as type ip. Note that you cannot change the mapping (including the type) of an existing field. What you need to do if you want to change the mapping of existing documents is reindex those documents to another index with the updated mapping.

So, first create a new index, which you create with the new mapping. Then use the reindex API to get all documents from the old index A into the new index B. As those documents get reindexed, they will get the updated mapping applied to them. More info about the reindex API here: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html

5

I remember I had the type set to ip. Let me clean everything up and start over again and re-confirm it.

6

Hello Abdon,

Here is what I tried in the mentioned sequence.
Query: GET /hostaddressindex/host/_mapping
Response:
{
"hostaddressindex": {
"mappings": {
"host": {
"_all": {
"enabled": false
},
"properties": {
"ipAddress": {
"type": "ip",
"fields": {
"raw": {
"type": "text"
}
}
},
"link": {
"type": "text",
"index": false
},
"timeInterval": {
"type": "integer",
"index": false
}
}
}
}
}
}

Query: POST /hostaddressindex/host/1
{
"ipAddress" : "192.168.1.1",
"link" : "192.168.1.1|3",
"timeInterval" : [1]
}

Response:
{
"_index": "hostaddressindex",
"_type": "host",
"_id": "1",
"_version": 4,
"result": "updated",
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"created": false
}

Query:GET /hostaddressindex/host/1
Response:
{
"_index": "hostaddressindex",
"_type": "host",
"_id": "1",
"_version": 4,
"found": true,
"_source": {
"ipAddress": "192.168.1.1",
"link": "192.168.1.1|3",
"timeInterval": [
1
]
}
}

Query:
GET _search
{
"aggs" : {
"ip_ranges" : {
"ip_range" : {
"field" : "ipAddress.raw",
"ranges" : [
{
"mask" : "192.168.0.0/16"
}
]
}
}
}
}

Response:
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 10,
"successful": 5,
"failed": 5,
"failures": [
{
"shard": 0,
"index": "indexing",
"node": "qzkSbw6dTDugoLnF5mECKw",
"reason": {
"type": "illegal_argument_exception",
"reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [ipAddress] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
}
}
]
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "hostaddressindex",
"_type": "host",
"_id": "1",
"_score": 1,
"_source": {
"ipAddress": "192.168.1.1",
"link": "192.168.1.1|3",
"timeInterval": [
1
]
}
}
]
},
"aggregations": {
"ip_ranges": {
"buckets": [
{
"key": "192.168.0.0/16",
"from": "192.168.0.0",
"to": "192.169.0.0",
"doc_count": 1
}
]
}
}
}

7

Instead of running the aggregation against ipAddress.raw (which is mapped as type text), you need to run it against ipAddress (which is mapped as type ip). The following should work now:

GET _search
{
  "size": 0,
  "aggs": {
    "ip_ranges": {
      "ip_range": {
        "field": "ipAddress",
        "ranges": [
          {
            "mask": "192.168.0.0/16"
          }
        ]
      }
    }
  }
}
8

I am sorry I pasted the wrong thing. It is ipAddress instead of ipAddress.raw. Still I get the issue.This is the query I fired:
GET _search
{
"aggs" : {
"ip_ranges" : {
"ip_range" : {
"field" : "ipAddress",
"ranges" : [
{
"mask" : "192.168.0.0/16"
}
]
}
}
}
}

9

Can you please post the output of:

GET /hostaddressindex/_mapping

10

{
"hostaddressindex": {
"mappings": {
"server": {
"_all": {
"enabled": false
},
"properties": {
"ipAddress": {
"type": "ip",
"fields": {
"raw": {
"type": "text"
}
}
},
"link": {
"type": "text",
"index": false
},
"timeInterval": {
"type": "integer",
"index": false
}
}
},
"client": {
"_all": {
"enabled": false
},
"properties": {
"ipAddress": {
"type": "ip",
"fields": {
"raw": {
"type": "text"
}
}
},
"link": {
"type": "text",
"index": false
},
"timeInterval": {
"type": "integer",
"index": false
}
}
},
"host": {
"_all": {
"enabled": false
},
"properties": {
"ipAddress": {
"type": "ip",
"fields": {
"raw": {
"type": "text"
}
}
},
"link": {
"type": "text",
"index": false
},
"query": {
"properties": {
"match_all": {
"type": "object"
}
}
},
"timeInterval": {
"type": "integer",
"index": false
}
}
}
}
}
}

11

The error is coming from an index named indexing which you are searching because you are doing GET _search rather than GET /hostaddressindex/_search. The index indexing probably also has a field call ipAddress but that one is mapping to text. You need to search just the index you want the data from, or if you want data from both indexes you will need to re-index indexing with the ipAddress field mapping as the ip type

1 Like
12

My bad...I didn't realise this and just skipped the indexing index in the error.

Thanks Colin and Abdon

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.