# ML - Datafeed is encountering errors extracting data: all shards failed

**URL:** <https://discuss.elastic.co/t/ml-datafeed-is-encountering-errors-extracting-data-all-shards-failed/264994>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-machine-learning\
**Created:** [February 21, 2021, 4:59pm UTC](https://discuss.elastic.co/t/ml-datafeed-is-encountering-errors-extracting-data-all-shards-failed/264994 "2021-02-21T16:59:09Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![cezar996](https://avatars.discourse-cdn.com/v4/letter/c/e95f7d/32.png) [@cezar996](https://discuss.elastic.co/u/cezar996)\
**Post date:** [February 21, 2021, 4:59pm UTC](https://discuss.elastic.co/t/ml-datafeed-is-encountering-errors-extracting-data-all-shards-failed/264994/1 "2021-02-21T16:59:09Z")

</div>

Hi!

I have created a ML job, but when I select `Start Datafeed`, I get this error: `Datafeed is encountering errors extracting data: all shards failed`, as in the image below.

 ![Screenshot from 2021-02-21 08-50-42](https://us1.discourse-cdn.com/elastic/original/3X/a/f/afd1d3d362b49ba165fa2ac20d2ae887cb6e50cb.png)

I really dont have any idea why I get this. I have used the same configuration some months ago and it worked with no problems. But today it doesn't work at all.

If you can give me a piece of advice for solving this problem, I would appreciate a lot.

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [February 21, 2021, 5:32pm UTC](https://discuss.elastic.co/t/ml-datafeed-is-encountering-errors-extracting-data-all-shards-failed/264994/2 "2021-02-21T17:32:27Z")

</div>

What is the datafeed configuration?

(it would be the output of the following)

```auto
GET _ml/datafeeds/datafeed-dns_exfiltration

```

Is the index pattern referenced in the datafeed configuration queryable?

```auto
GET yourindexname/_search

```

(where `yourindexname` is whatever is found in the `indices` section of the response from the first command)

---

<div class="post-metadata">

**Author:** ![cezar996](https://avatars.discourse-cdn.com/v4/letter/c/e95f7d/32.png) [@cezar996](https://discuss.elastic.co/u/cezar996)\
**Post date:** [February 21, 2021, 6:20pm UTC](https://discuss.elastic.co/t/ml-datafeed-is-encountering-errors-extracting-data-all-shards-failed/264994/3 "2021-02-21T18:20:49Z")

</div>

Hello @richcollier! It is a pleasure for me to meet you again! 🙂 You helped me in some previous posts and thank you a lot. You are a guru in ELK ML 🙂

To respond to your questions, for the command `GET _ml/datafeeds/datafeed-dns_exfiltration`, I get:

```
{
  "count" : 1,
  "datafeeds" : [
    {
      "datafeed_id" : "datafeed-dns_exfiltration",
      "job_id" : "dns_exfiltration",
      "query_delay" : "65004ms",
      "indices" : [
        "packetbeat-*"
      ],
      "query" : {
        "bool" : {
          "should" : [
            {
              "match_phrase" : {
                "type" : "dns"
              }
            }
          ],
          "minimum_should_match" : 1,
          "filter" : [
            {
              "match_phrase" : {
                "type" : "dns"
              }
            }
          ],
          "must_not" : []
        }
      },
      "script_fields" : {
        "hrd" : {
          "script" : {
            "source" : "return domainSplit(doc['dns.question.name'].value).get(1);",
            "lang" : "painless"
          },
          "ignore_failure" : false
        },
        "sub" : {
          "script" : {
            "source" : "return domainSplit(doc['dns.question.name'].value).get(0);",
            "lang" : "painless"
          },
          "ignore_failure" : false
        }
      },
      "scroll_size" : 1000,
      "chunking_config" : {
        "mode" : "auto"
      },
      "delayed_data_check_config" : {
        "enabled" : true
      }
    }
  ]
}

```

As regards this command, `GET packetbeat-*/_search`, I obtain:

```
{
  "took" : 17,
  "timed_out" : false,
  "_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
  },
  "hits" : {
"total" : {
  "value" : 1128,
  "relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
  {
    "_index" : "packetbeat-7.6.2",
    "_type" : "_doc",
    "_id" : "mHtwxXcBN1tfp17DWxnv",
    "_score" : 1.0,
    "_source" : {
      "@timestamp" : "2021-02-21T16:33:30.001Z",
      "network" : {
        "bytes" : 1099865,
        "packets" : 4023,
        "type" : "ipv4",
        "transport" : "tcp",
        "community_id" : "1:YqII7mbgOpdT5mf3MMJdPvr+dhk="
      },
      "host" : {
        "name" : "ubuntu",
        "hostname" : "ubuntu",
        "architecture" : "x86_64",
        "os" : {
          "codename" : "bionic",
          "platform" : "ubuntu",
          "version" : "18.04.1 LTS (Bionic Beaver)",
          "family" : "debian",
          "name" : "Ubuntu",
          "kernel" : "4.15.0-29-generic"
        },
        "id" : "8f68089f99fc4e6db58b1d98c7ee3d64",
        "containerized" : false
      },
      "ecs" : {
        "version" : "1.4.0"
      },
      "type" : "flow",
      "source" : {
        "bytes" : 786344,
        "ip" : "127.0.0.1",
        "port" : 58526,
        "packets" : 2408
      },
      "destination" : {
        "packets" : 1615,
        "ip" : "127.0.0.1",
        "port" : 9200,
        "bytes" : 313521
      },
      "event" : {
        "end" : "2021-02-21T16:33:29.752Z",
        "duration" : 2627491892158,
        "dataset" : "flow",
        "kind" : "event",
        "category" : "network_traffic",
        "action" : "network_flow",
        "start" : "2021-02-21T15:49:42.261Z"
      },
      "agent" : {
        "type" : "packetbeat",
        "ephemeral_id" : "00bfd7ae-a87e-4c49-9a33-0e46bea2b072",
        "hostname" : "ubuntu",
        "id" : "b306d0ba-7d77-4c5b-a009-a2bcb70c4922",
        "version" : "7.6.2"
      },
      "flow" : {
        "id" : "EAT/////AP//////CP8AAAF/AAABfwAAAZ7k8CM",
        "final" : false
      }
    }
  },
  ....... and so on
  }
}
```

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [February 22, 2021, 11:33am UTC](https://discuss.elastic.co/t/ml-datafeed-is-encountering-errors-extracting-data-all-shards-failed/264994/4 "2021-02-22T11:33:04Z")

</div>

thanks for that info. At least we know the index exists and that it is queryable. Let's now validate that the type of query this ML job is configured for will also execute. Can you try to see if the following query returns without problems/errors?

```auto
GET packetbeat-*/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "match_phrase": {
            "type": "dns"
          }
        }
      ],
      "minimum_should_match": 1,
      "filter": [
        {
          "match_phrase": {
            "type": "dns"
          }
        }
      ],
      "must_not": []
    }
  },
  "script_fields": {
    "hrd": {
      "script": {
        "source": "return domainSplit(doc['dns.question.name'].value).get(1);",
        "lang": "painless"
      },
      "ignore_failure": false
    },
    "sub": {
      "script": {
        "source": "return domainSplit(doc['dns.question.name'].value).get(0);",
        "lang": "painless"
      },
      "ignore_failure": false
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [February 22, 2021, 11:36am UTC](https://discuss.elastic.co/t/ml-datafeed-is-encountering-errors-extracting-data-all-shards-failed/264994/5 "2021-02-22T11:36:51Z")

</div>

If that does work, by the way, then there must be some other thing logistically wrong and we would need to look more closely at the elasticsearch.log file for detailed errors. To do that, probably the easiest way would be to clone the existing job and force it to run over some historical data immediately. That way, you can look in the elasticsearch.log file for the errors that are incurring when you forced it to run.

---

<div class="post-metadata">

**Author:** ![cezar996](https://avatars.discourse-cdn.com/v4/letter/c/e95f7d/32.png) [@cezar996](https://discuss.elastic.co/u/cezar996)\
**Post date:** [February 22, 2021, 1:04pm UTC](https://discuss.elastic.co/t/ml-datafeed-is-encountering-errors-extracting-data-all-shards-failed/264994/6 "2021-02-22T13:04:51Z")

</div>

When I run this, I get an error:

```
{
  "error" : {
    "root_cause" : [
      {
        "type" : "script_exception",
        "reason" : "runtime error",
        "script_stack" : [
          "org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:762)",
          "org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",
          "org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:311)",
          "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:101)",
          "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:98)",
          "java.base/java.security.AccessController.doPrivileged(AccessController.java:312)",
          "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:98)",
          "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)",
          "return domainSplit(doc['dns.question.name'].value).get(1);",
          " ^---- HERE"
        ],
        "script" : "return domainSplit(doc['dns.question.name'].value).get(1);",
        "lang" : "painless"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "packetbeat-7.6.2",
        "node" : "bdCh8LxiRIu9AuIEshOKdg",
        "reason" : {
          "type" : "script_exception",
          "reason" : "runtime error",
          "script_stack" : [
            "org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:762)",
            "org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",
            "org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:311)",
            "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:101)",
            "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:98)",
            "java.base/java.security.AccessController.doPrivileged(AccessController.java:312)",
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:98)",
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)",
            "return domainSplit(doc['dns.question.name'].value).get(1);",
            " ^---- HERE"
          ],
          "script" : "return domainSplit(doc['dns.question.name'].value).get(1);",
          "lang" : "painless",
          "caused_by" : {
            "type" : "illegal_argument_exception",
            "reason" : "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [dns.question.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
          }
        }
      }
    ]
  },
  "status" : 400
}

```

Do you know how could I solve this?

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [February 22, 2021, 2:15pm UTC](https://discuss.elastic.co/t/ml-datafeed-is-encountering-errors-extracting-data-all-shards-failed/264994/7 "2021-02-22T14:15:20Z")

</div>

The field `dns.question.name` needs to be of type `keyword` for it to comply with ECS (see [DNS Fields | Elastic Common Schema (ECS) Reference [1.8] | Elastic](https://www.elastic.co/guide/en/ecs/current/ecs-dns.html#field-dns-question-name)). You can confirm what type it is set to in your index via:

```auto
GET packetbeat-*/_mapping/field/dns.question.name

```

If you ingested the data via packetbeat, it should have done this for you - I'm not sure why it wouldn't have. How are you ingesting this data?

---

<div class="post-metadata">

**Author:** ![cezar996](https://avatars.discourse-cdn.com/v4/letter/c/e95f7d/32.png) [@cezar996](https://discuss.elastic.co/u/cezar996)\
**Post date:** [February 22, 2021, 2:21pm UTC](https://discuss.elastic.co/t/ml-datafeed-is-encountering-errors-extracting-data-all-shards-failed/264994/8 "2021-02-22T14:21:27Z")

</div>

Hi @richcollier!

After running the above GET command, I obtained this:

```
{
  "packetbeat-7.6.2-2021.02.22-000001" : {
    "mappings" : {
      "dns.question.name" : {
        "full_name" : "dns.question.name",
        "mapping" : {
          "name" : {
            "type" : "keyword",
            "ignore_above" : 1024
          }
        }
      }
    }
  }
}

```

Yes, all the data is ingested via packetbeat.

---

<div class="post-metadata">

**Author:** ![cezar996](https://avatars.discourse-cdn.com/v4/letter/c/e95f7d/32.png) [@cezar996](https://discuss.elastic.co/u/cezar996)\
**Post date:** [February 22, 2021, 2:24pm UTC](https://discuss.elastic.co/t/ml-datafeed-is-encountering-errors-extracting-data-all-shards-failed/264994/9 "2021-02-22T14:24:35Z")

</div>

However, before getting the error with `all shards failed` for the job with DNS Data Exfiltration, I got this one:

`"{\"error\":{\"root_cause\":[{\"type\":\"status_exception\",\"reason\":\"Could not open job because no ML nodes with sufficient capacity were found\"}],\"type\":\"status_exception\",\"reason\":\"Could not open job because no ML nodes with sufficient capacity were found\",\"caused_by\":{\"type\":\"illegal_state_exception\",\"reason\":\"Could not open job because no suitable nodes were found, allocation explanation [Not opening job [dns_exfiltration] on node [{ubuntu}{ml.machine_memory=4112064512}{ml.max_open_jobs=20}], because this node has insufficient available memory. Available memory for ML [1233619353], memory required by existing jobs [1115684864], estimated memory required for this job [1084227584]]\"}},\"status\":429}"`

Also I have no idea, how I could solve this...

---

<div class="post-metadata">

**Author:** ![cezar996](https://avatars.discourse-cdn.com/v4/letter/c/e95f7d/32.png) [@cezar996](https://discuss.elastic.co/u/cezar996)\
**Post date:** [February 22, 2021, 2:30pm UTC](https://discuss.elastic.co/t/ml-datafeed-is-encountering-errors-extracting-data-all-shards-failed/264994/10 "2021-02-22T14:30:15Z")

</div>

After that, I have tried to create a new job for HTTP Data Exfiltration, using the GitHub Security Analytics Recipes, but when I pressed `Start Datafeed`, I got some errors as regards mapping. In the example below, the error is for `http.request.headers.host`, but I also got for `bytes_in`:

`"{\"error\":{\"root_cause\":[{\"type\":\"status_exception\",\"reason\":\"[datafeed-http_data_exfiltration] cannot retrieve field [http.request.headers.host] because it has no mappings\"}],\"type\":\"status_exception\",\"reason\":\"[datafeed-http_data_exfiltration] cannot retrieve field [http.request.headers.host] because it has no mappings\"},\"status\":400}"`

It is like I am in a chain of errors 😔

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [February 22, 2021, 5:08pm UTC](https://discuss.elastic.co/t/ml-datafeed-is-encountering-errors-extracting-data-all-shards-failed/264994/11 "2021-02-22T17:08:29Z")

</div>

Your query results imples that the `packetbeat-*` index pattern is matching an index called `packetbeat-7.6.2-2021.02.22-000001` which has the right mappings, but I wonder if there are other, older indices that match the pattern but don't have the right mappings (??). Can you verify how many indices match the index pattern?

```auto
GET _cat/indices/packetbeat-*

```

---

<div class="post-metadata">

**Author:** ![cezar996](https://avatars.discourse-cdn.com/v4/letter/c/e95f7d/32.png) [@cezar996](https://discuss.elastic.co/u/cezar996)\
**Post date:** [February 22, 2021, 9:12pm UTC](https://discuss.elastic.co/t/ml-datafeed-is-encountering-errors-extracting-data-all-shards-failed/264994/12 "2021-02-22T21:12:41Z")

</div>

I solved this problem. It was because I forgot to put `send_all_headers: true` for `packetbeat.protocols.http`, in the` packetbeat.yml` file.

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [February 23, 2021, 11:19am UTC](https://discuss.elastic.co/t/ml-datafeed-is-encountering-errors-extracting-data-all-shards-failed/264994/13 "2021-02-23T11:19:51Z")

</div>

The insufficient capacity message is telling you that your 4GB system is too small for the job you're asking ML to execute (which might take around 1GB on its own). The amount of memory ML is allowed to use of the machine is governed by settings shown in: [Machine learning settings in Elasticsearch | Elasticsearch Reference [7.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-settings.html)

Specifically, `xpack.ml.max_machine_memory_percent` which defaults to 30% Therefore a 4GB node will allow about 1.2GB to be allocated to ML. Since you already have about 1.1GB being allocated to existing jobs (according to the error message):

```auto
node [{ubuntu}{ml.machine_memory=4112064512}{ml.max_open_jobs=20}], because this node has insufficient available memory. Available memory for ML [1233619353], memory required by existing jobs [1115684864], estimated memory required for this job [1084227584]]\"}}

```

...then you cannot open up any more jobs that require a lot of memory to run. Solutions:

1. Get a bigger node
2. Delete other jobs
3. Slightly increase `xpack.ml.max_machine_memory_percent`, but be careful. ML operates outside the JVM heap. You need RAM for the JVM, ML, core Linux, and anything else running on that node.

---

<div class="post-metadata">

**Author:** ![cezar996](https://avatars.discourse-cdn.com/v4/letter/c/e95f7d/32.png) [@cezar996](https://discuss.elastic.co/u/cezar996)\
**Post date:** [February 23, 2021, 4:21pm UTC](https://discuss.elastic.co/t/ml-datafeed-is-encountering-errors-extracting-data-all-shards-failed/264994/14 "2021-02-23T16:21:13Z")

</div>

Hi @richcollier! Thank you for your response. I finally uninstalled elasticsearch, installed it again and things seem to work fine. However, I tried to create another job for suspicious process activity, as described in the Github Security Analytics Recipes:

```
PUT _ml/anomaly_detectors/suspicious_process_activity
{
  "description": "Suspicious Process Activity",
  "analysis_config": {
    "bucket_span": "5m",
    "influencers": [
      "auditd.log.a0",
      "host.name"
    ],
    "detectors": [
      {
        "function": "rare",
        "by_field_name": "auditd.log.a0",
        "partition_field_name": "host.name"
      }
    ]
  },
  "data_description": {
    "time_field": "@timestamp",
    "time_format": "epoch_ms"
  },
  "model_plot_config": {
      "enabled" : true
  }
}

```

and

```
PUT _ml/datafeeds/datafeed-suspicious_process_activity
{
  "job_id": "suspicious_process_activity",
  "indices": [
    "filebeat-*"
  ],
  "query": {
    "term": {
      "event.action": {
        "value": "EXECVE"
      }
    }
  },
  "query_delay": "60s",
  "frequency": "300s",
  "scroll_size": 1000
}

```

I queried Kibana this way, ` "event.action": "EXECVE"` and got results as in the image below:

 ![Screenshot from 2021-02-23 18-18-06](https://us1.discourse-cdn.com/elastic/original/3X/d/0/d0c16828a557e0ab5f8fda7a15f0641f8803617a.png)

However, I got the following error, `Datafeed lookback retrieved no data` and dont understand why ☹

 ![Screenshot from 2021-02-23 17-30-40](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ae431ed114830d0fe4d1365d71e781ec2a06085.png)

Can you help me, please?

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [February 23, 2021, 5:35pm UTC](https://discuss.elastic.co/t/ml-datafeed-is-encountering-errors-extracting-data-all-shards-failed/264994/15 "2021-02-23T17:35:33Z")

</div>

Certainly nothing obvious from the job or datafeed config. How did you start the datafeed? Did you have it look back over any historical data?

---

<div class="post-metadata">

**Author:** ![cezar996](https://avatars.discourse-cdn.com/v4/letter/c/e95f7d/32.png) [@cezar996](https://discuss.elastic.co/u/cezar996)\
**Post date:** [February 23, 2021, 6:52pm UTC](https://discuss.elastic.co/t/ml-datafeed-is-encountering-errors-extracting-data-all-shards-failed/264994/16 "2021-02-23T18:52:06Z")

</div>

In the last image I have posted, I have pressed on the 3 dots of the `suspicious_process_activity` job and I selected `Start Datafeed`. In the window that appeared, I have selected from the beginning of data until now (real-time search).

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [February 23, 2021, 7:27pm UTC](https://discuss.elastic.co/t/ml-datafeed-is-encountering-errors-extracting-data-all-shards-failed/264994/17 "2021-02-23T19:27:13Z")

</div>

Ok - that's good to know. So now you need to systematically figure out why you're not getting data. Again, like we did earlier, you need to simulate the query that the datafeed is doing against the raw data, as in:

```auto
GET filebeat-*/_search
{
  "query": {
    "term": {
      "event.action": {
        "value": "EXECVE"
      }
    }
  }
}

```

and make sure that the query yields results, and those results contain the fields that you care about (in this case, `auditd.log.a0` and `host.name`)

I know that you tried in Kibana, but I prefer to do it via the `_search` API and use the exact query that the datafeed uses to eliminate variables

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [February 23, 2021, 7:28pm UTC](https://discuss.elastic.co/t/ml-datafeed-is-encountering-errors-extracting-data-all-shards-failed/264994/18 "2021-02-23T19:28:09Z")

</div>

Out of curiosity, if you've re-installed the stack from scratch, why aren't you using v7.11?

---

<div class="post-metadata">

**Author:** ![cezar996](https://avatars.discourse-cdn.com/v4/letter/c/e95f7d/32.png) [@cezar996](https://discuss.elastic.co/u/cezar996)\
**Post date:** [February 23, 2021, 9:30pm UTC](https://discuss.elastic.co/t/ml-datafeed-is-encountering-errors-extracting-data-all-shards-failed/264994/19 "2021-02-23T21:30:50Z")

</div>

Hi @richcollier! Thank you very much for your explanations! You nicely exposed to me the necessary steps to debug the problem. After running the GET command inside the Dev Tools section of Kibana, I got the following result:

```
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : []
  }
}

```

You were right that something may be strange here, although in Kibana I got results. 👏 So, it seems that here there are some problems.

---

<div class="post-metadata">

**Author:** ![cezar996](https://avatars.discourse-cdn.com/v4/letter/c/e95f7d/32.png) [@cezar996](https://discuss.elastic.co/u/cezar996)\
**Post date:** [February 23, 2021, 9:42pm UTC](https://discuss.elastic.co/t/ml-datafeed-is-encountering-errors-extracting-data-all-shards-failed/264994/20 "2021-02-23T21:42:41Z")

</div>

Yes, a very good question. I have used ELK v.7.6.2 some months ago when I tried to detect DNS Data Exfiltration, but in a few days I will have a presentation at my university where I will show how anomalies can be detected using ELK. Giving the fact that there remained just a few days until presentation, I preferred to use something I already know and have worked, then something new which could have required some time to get accustomed to.

[Next page](https://discuss.elastic.co/t/ml-datafeed-is-encountering-errors-extracting-data-all-shards-failed/264994.md?page=2)