Anomaly detection / ML Setup

[tsp]

When setting up the ML within the Anomaly detection (Stack 11.1.2) , it says the following:

At least one index matching logs-pattern-* has a field called event.dataset without the correct type.

Here is a sample record:

{
  "_index": "cen-logs-2021.05.07",
  "_type": "_doc",
  "_id": "y-mkRXkBxSJkcJbaDJAK",
  "_version": 1,
  "_score": null,
  "_source": {
    "time": "2021-05-07T07:04:08.45951746Z",
    "@version": "1",
    "host": {
      "name": "XXXX.YYYY.ZZZZ.net",
      "hostname": "XXXX.YYYY.ZZZZ.net"
    },
    "@timestamp_fluentbit": 1620371048.459517,
    "ecs": {
      "version": "1.9.0"
    },
    "topickey": "a_topic",
    "message": "2021-05-07T07:04:08.459Z\tERROR\t[kafka]\tkafka/client.go:317\tKafka (topic=a_topic): kafka: client has run out of available brokers to talk to (Is your cluster reachable?)\n",
    "clustername": AAAA.XXXX.YYYY.ZZZZ.net",
    "@timestamp": "2021-05-07T07:04:09.304Z",
    "kubernetes": {
      "pod_id": "7e925054-a7df-11eb-b927-94f128a205f0",
      "container_name": "heartbeat",
      "host": "XXXX.YYYY.ZZZZ.net",
      "container_hash": "BBBB-container-registry.docker.YYYY.net/elastic/heartbeat@sha256:14a150d5c4430076477860715faf0fb63e0c838e021c3dfd28ddda6710c043de",
      "namespace_name": "XXXX-acc",
      "pod_name": "heartbeat-deployment-f67d8d945-qgjqz",
      "docker_id": "31d306622560865f2cb6d5f628971f27d3e58ab597e59e3319504cd98909796f",
      "container_image": "BBBB-container-registry.docker.XXXX.net/elastic/heartbeat@sha256:14a150d5c4430076477860715faf0fb63e0c838e021c3dfd28ddda6710c043de",
      "pod": {
        "id": "7e925054-a7df-11eb-b927-94f128a205f0"
      },
      "labels": {
        "release": "RELEASE-NAME",
        "pod-template-hash": "923848501",
        "chart": "heartbeat-7.11.2",
        "app_": "RELEASE-NAME-heartbeat-metrics"
      },
      "annotations": {
        "configChecksum": "19b27e247c38d2781ca94daa41c6cf6e146dcec527004438e7280e9b5ee65c4",
        "openshift.io/scc": "restricted",
        "loggingkafkatopic": "a_topc"
      }
    },
    "stream": "stderr",
    "log": {
      "file": {
        "path": "/Container/log"
      }
    },
    "container": {
      "id": "AAAA-container-registry.docker.XXXX.net/elastic/heartbeat@sha256:14a150d5c4430076477860715faf0fb63e0c838e021c3dfd28ddda6710c043de"
    },
    "event": {
      "kind": "event",
      "type": "info",
      "category": "host",
      "dataset": "module.dataset"
    }
  },
  "fields": {
    "@timestamp": [
      "2021-05-07T07:04:09.304Z"
    ],
    "time": [
      "2021-05-07T07:04:08.459Z"
    ]
  },
  "sort": [
    1620371049304
  ]
}

Any hint appreciated

Thanks

[richcollier]

@tsp

  "_index": "cen-logs-2021.05.07",

Are you enabling ML from the Logs UI?

Judging by your error message and your sample document, it looks like your index of cen-logs-2021.05.07 doesn't match the default index patterns in the Logs UI (or you may have logs in some other indices that do match the pattern, but don't conform to having the required event.dataset field).

[tsp]

Thanks, I was indeed enabling it through the UI.

I found out afterwards that some data was shipped into a complete different format, now that both formats are handled, it works as a charm.
Thanks.