Detector field "clientIP.keyword" is not an aggregatable field

Elastic Stack Elasticsearch
1

Hello,

Stack version : 7.9.3

I found issue when I tried to create a machine learning job in kibana

The error : Detector field "clientIP.keyword" is not an aggregatable field

The data looks like this in kibana

{
  "_index": "sample-dns-2021.03.07",
  "_type": "_doc",
  "_id": "71HPC3gBgsMd4G48HpLZ",
  "_version": 1,
  "_score": null,
  "_source": {
    "service": "tmm",
    "requestLn": 45,
    "origin": "CACHE",
    "location": "AS ID Nama Kota ",
    "questionName": "time-a.timefreq.bldrdoc.gov",
    "answer": "{time-a.timefreq.bldrdoc.gov.\t295\tIN\tCNAME\ttime-a-b.nist.gov }",
    "response": "DNS_RESPONSE",
    "rulePath": "/Common/log_dns_query_2",
    "clientIP": "103.124.89.250",
    "clientDescription": "Testing only",
    "queryTime": 0,
    "responseLn": 119,
    "@timestamp": "2021-03-07T08:30:26.504Z",
    "pid": "27589",
    "listener": "1.1.1.1",
    "host": "192.68.1.1",
    "message": "xxx",
    "@version": "1",
    "logLevel": "info",
    "questionType": "AAAA",
    "deviceTimestamp": "Mar  7 15:30:26",
    "deviceHostname": "GTM Bla Bla"
  },
  "fields": {
    "@timestamp": [
      "2021-03-07T08:30:26.504Z"
    ]
  },
  "sort": [
    1615105826504
  ]
}

Is there something wrong with field ClientIP that make it not aggregatable? I already using this field in my visualization and it works just fine.

Thank you

2

What's the mapping?

3

Here is the mapping

{
  "sample-dns-2021.03.07" : {
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "@version" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "answer" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "clientDescription" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "clientIP" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "host" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "ldns" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "listener" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "location" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "port" : {
          "type" : "long"
        },
        "queryTime" : {
          "type" : "long"
        },
        "questionName" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "questionType" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "request" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "request1" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "request2" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "requestLn" : {
          "type" : "integer",
          "ignore_malformed" : false,
          "coerce" : true
        },
        "requestMethod" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "response" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "responseCode" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "responseLn" : {
          "type" : "integer",
          "ignore_malformed" : false,
          "coerce" : true
        },
        "tags" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "version" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    }
  }
}
4

Why clientIP is not a IP datatype?

I believe that it would be better to make it an IP.

Not sure if that will answer though.

Same for host and listener BTW.

5

Yes it doesnt help

What I dont understand I can create machine learning job with my other index

6

Here is the mapping of my other index

The detector field is source and contain also ip address

{
  "msrouter-2021.01.18" : {
    "mappings" : {
      "properties" : {
        
        "deviceHostname" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "deviceMessage" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "deviceVendor" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "facility" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "facilityMnemonic" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "host" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "logLevel" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "message" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "reason" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "source" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "tags" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "user" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    }
  }
}
7

I don't understand either. May be someone from the machine learning team could help?

8

First you need to check for consistent mappings across every index which matches the pattern used in the data feed; if you are using wildcards check there aren't different mappings for clientIP.keyword in some indices. However, note that this error message can be misleadingly reported for other reasons. See this issue.

2 Likes
9

Yes looks like misleading error

I can create the job using ML APIs now

Thanks for the info

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.