Using bucket aggregation on IPs not giving expected results


(Sambit Kabi) #1

Hello,

I am trying to search for documents containing an IP which falls under a range. As specified in the image the docs have IP 192.168.255.100 which falls under the mask: 192.168.0.0/24. But the result that I get is not as per my understanding. I was expecting to get docs in the bucket. Moreover if I try with 192.168.0.0/24 I get ArrayOutOfBoundIndexException. Not sure what's happening. Also from and to string is weird with some different
encoding.

Could you please point me out as to where its getting wrong?

Thanks


(Abdon Pijpelink) #2

The name of the field that you run this aggregation against (ipAddress.keyword) suggests that you are running this aggregation against a field that has been mapped as type keyword. You need to map this field as type ip if you want to use the ip_range aggregation.


(Sambit Kabi) #3

If I use ip type in aggregation it gives me error saying "Fielddata is disabled on text fields by default. Set fielddata=true on [ipAddress] in order to load fielddata in memory by uninverting the inverted index"

I don't want to use fielddata because it would be costly in terms of memory. So I opted for multi-field and used keyword subfield to use in aggs.


(Abdon Pijpelink) #4

I think your field ipAddress is actually mapped as a type text rather than as type ip. You could check that by retrieving the mapping:

GET hostaddressindex/client/_mapping

I think you will see "type": "text" instead of "type": "ip" for the ipAddress field.

You will have to change your mapping such that ipAddress is mapped as type ip. Note that you cannot change the mapping (including the type) of an existing field. What you need to do if you want to change the mapping of existing documents is reindex those documents to another index with the updated mapping.

So, first create a new index, which you create with the new mapping. Then use the reindex API to get all documents from the old index A into the new index B. As those documents get reindexed, they will get the updated mapping applied to them. More info about the reindex API here: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html


(Sambit Kabi) #5

I remember I had the type set to ip. Let me clean everything up and start over again and re-confirm it.


(Sambit Kabi) #6

Hello Abdon,

Here is what I tried in the mentioned sequence.
Query: GET /hostaddressindex/host/_mapping
Response:
{
"hostaddressindex": {
"mappings": {
"host": {
"_all": {
"enabled": false
},
"properties": {
"ipAddress": {
"type": "ip",
"fields": {
"raw": {
"type": "text"
}
}
},
"link": {
"type": "text",
"index": false
},
"timeInterval": {
"type": "integer",
"index": false
}
}
}
}
}
}

Query: POST /hostaddressindex/host/1
{
"ipAddress" : "192.168.1.1",
"link" : "192.168.1.1|3",
"timeInterval" : [1]
}

Response:
{
"_index": "hostaddressindex",
"_type": "host",
"_id": "1",
"_version": 4,
"result": "updated",
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"created": false
}

Query:GET /hostaddressindex/host/1
Response:
{
"_index": "hostaddressindex",
"_type": "host",
"_id": "1",
"_version": 4,
"found": true,
"_source": {
"ipAddress": "192.168.1.1",
"link": "192.168.1.1|3",
"timeInterval": [
1
]
}
}

Query:
GET _search
{
"aggs" : {
"ip_ranges" : {
"ip_range" : {
"field" : "ipAddress.raw",
"ranges" : [
{
"mask" : "192.168.0.0/16"
}
]
}
}
}
}

Response:
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 10,
"successful": 5,
"failed": 5,
"failures": [
{
"shard": 0,
"index": "indexing",
"node": "qzkSbw6dTDugoLnF5mECKw",
"reason": {
"type": "illegal_argument_exception",
"reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [ipAddress] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
}
}
]
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "hostaddressindex",
"_type": "host",
"_id": "1",
"_score": 1,
"_source": {
"ipAddress": "192.168.1.1",
"link": "192.168.1.1|3",
"timeInterval": [
1
]
}
}
]
},
"aggregations": {
"ip_ranges": {
"buckets": [
{
"key": "192.168.0.0/16",
"from": "192.168.0.0",
"to": "192.169.0.0",
"doc_count": 1
}
]
}
}
}


(Abdon Pijpelink) #7

Instead of running the aggregation against ipAddress.raw (which is mapped as type text), you need to run it against ipAddress (which is mapped as type ip). The following should work now:

GET _search
{
  "size": 0,
  "aggs": {
    "ip_ranges": {
      "ip_range": {
        "field": "ipAddress",
        "ranges": [
          {
            "mask": "192.168.0.0/16"
          }
        ]
      }
    }
  }
}

(Sambit Kabi) #8

I am sorry I pasted the wrong thing. It is ipAddress instead of ipAddress.raw. Still I get the issue.This is the query I fired:
GET _search
{
"aggs" : {
"ip_ranges" : {
"ip_range" : {
"field" : "ipAddress",
"ranges" : [
{
"mask" : "192.168.0.0/16"
}
]
}
}
}
}


(Abdon Pijpelink) #9

Can you please post the output of:

GET /hostaddressindex/_mapping


(Sambit Kabi) #10

{
"hostaddressindex": {
"mappings": {
"server": {
"_all": {
"enabled": false
},
"properties": {
"ipAddress": {
"type": "ip",
"fields": {
"raw": {
"type": "text"
}
}
},
"link": {
"type": "text",
"index": false
},
"timeInterval": {
"type": "integer",
"index": false
}
}
},
"client": {
"_all": {
"enabled": false
},
"properties": {
"ipAddress": {
"type": "ip",
"fields": {
"raw": {
"type": "text"
}
}
},
"link": {
"type": "text",
"index": false
},
"timeInterval": {
"type": "integer",
"index": false
}
}
},
"host": {
"_all": {
"enabled": false
},
"properties": {
"ipAddress": {
"type": "ip",
"fields": {
"raw": {
"type": "text"
}
}
},
"link": {
"type": "text",
"index": false
},
"query": {
"properties": {
"match_all": {
"type": "object"
}
}
},
"timeInterval": {
"type": "integer",
"index": false
}
}
}
}
}
}


(Colin Goodheart-Smithe) #11

The error is coming from an index named indexing which you are searching because you are doing GET _search rather than GET /hostaddressindex/_search. The index indexing probably also has a field call ipAddress but that one is mapping to text. You need to search just the index you want the data from, or if you want data from both indexes you will need to re-index indexing with the ipAddress field mapping as the ip type


(Sambit Kabi) #12

My bad...I didn't realise this and just skipped the indexing index in the error.

Thanks Colin and Abdon


(system) #13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.