# Understanding skipped documents in Data Frame Analytics

**URL:** <https://discuss.elastic.co/t/understanding-skipped-documents-in-data-frame-analytics/238760>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-machine-learning\
**Created:** [June 25, 2020, 9:44pm UTC](https://discuss.elastic.co/t/understanding-skipped-documents-in-data-frame-analytics/238760 "2020-06-25T21:44:50Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![davemoore](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davemoore/32/134233_2.png) [@davemoore](https://discuss.elastic.co/u/davemoore)\
**Post date:** [June 25, 2020, 9:44pm UTC](https://discuss.elastic.co/t/understanding-skipped-documents-in-data-frame-analytics/238760/1 "2020-06-25T21:44:50Z")

</div>

When running a Data Frame Analytics job in 7.8.0, documents with unsupported values are skipped ([source](https://www.elastic.co/guide/en/machine-learning/7.8/ml-dfa-limitations.html#dfa-missing-fields-limitations)).

Suppose I run a job that fails after the `loading_data` phase because every document was skipped, and the `data_counts` of the job is reported as:

`{"training_docs_count":0,"test_docs_count":0,"skipped_docs_count":1000}`

Is there a way to troubleshoot why a document was skipped during the job?

---

<div class="post-metadata">

**Author:** ![dmitri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dmitri/32/17997_2.png) [@dmitri](https://discuss.elastic.co/u/dmitri)\
**Post date:** [June 27, 2020, 8:17am UTC](https://discuss.elastic.co/t/understanding-skipped-documents-in-data-frame-analytics/238760/2 "2020-06-27T08:17:16Z")

</div>

Hi Dave,

Docs are skipped either because included fields have missing values and the analysis doesn't support missing values (ie. `outlier_detection`), or because they have a value the job cannot analyze (ie. an array with more than 1 element).

You can also use the [Explain API](https://www.elastic.co/guide/en/elasticsearch/reference/7.8/explain-dfanalytics.html) to see exactly which fields are included in the analysis and then check for which of them contain missing values (if you're using `outlier_detection` only) or arrays in your source index.

---

<div class="post-metadata">

**Author:** ![davemoore](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davemoore/32/134233_2.png) [@davemoore](https://discuss.elastic.co/u/davemoore)\
**Post date:** [June 30, 2020, 4:28pm UTC](https://discuss.elastic.co/t/understanding-skipped-documents-in-data-frame-analytics/238760/3 "2020-06-30T16:28:41Z")

</div>

Thanks @dmitri. I resolved the issue, and I believe it was caused by the number of analyzed fields in the documents.

My documents had 1,260 analyzed fields: one field for each 2-character permutation of the string `0123456789abcdefghijklmnopqrstuvwxyz`. Each field was a numerical count of the bigram (specifically a `long`) which was a valid data type for the job. All of those documents were skipped. When I reduced the the number analyzed fields to 36 (one field for each single character in the string) none of the documents were skipped and the job proceeded.

---

<div class="post-metadata">

**Author:** ![davemoore](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davemoore/32/134233_2.png) [@davemoore](https://discuss.elastic.co/u/davemoore)\
**Post date:** [July 28, 2020, 4:28pm UTC](https://discuss.elastic.co/t/understanding-skipped-documents-in-data-frame-analytics/238760/4 "2020-07-28T16:28:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.