# Data frame analytics on scripted fields ML possible?

**URL:** <https://discuss.elastic.co/t/data-frame-analytics-on-scripted-fields-ml-possible/285504>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-machine-learning\
**Created:** [September 29, 2021, 6:26pm UTC](https://discuss.elastic.co/t/data-frame-analytics-on-scripted-fields-ml-possible/285504 "2021-09-29T18:26:31Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![MLsuper](https://avatars.discourse-cdn.com/v4/letter/m/dfb087/32.png) [@MLsuper](https://discuss.elastic.co/u/MLsuper)\
**Post date:** [September 29, 2021, 6:26pm UTC](https://discuss.elastic.co/t/data-frame-analytics-on-scripted-fields-ml-possible/285504/1 "2021-09-29T18:26:31Z")

</div>

I am currently playing around with Machine learning on Kibana and I successfully learned how to use Anomaly detection to use scripted fields by editing the JSON.

Now I want to use data frame analytics specifically currently on regression and clicked advance editor since there are no scripted fields dropdown in my dependant variables drop down.

So I tried to add "script\_fields".... json query and it does not work using the query from

> [@Machine Learning on scripted field?](https://discuss.elastic.co/t/machine-learning-on-scripted-field/89054/2):
>
> Hello, Yes, creating a scripted\_field must be part of the datafeed configuration ([https://www.elastic.co/guide/en/x-pack/5.4/ml-put-datafeed.html](https://www.elastic.co/guide/en/x-pack/5.4/ml-put-datafeed.html)). To accomplish: Create an advanced job (only can do this technique on advanced jobs) Configure the job as you normally would. When creating detectors, instead of choosing an existing field, choose a name that we'll later assign to the script field. In this example, we are choosing the field name total\_error\_count which doesn't exist in our documen…

it works for Anomaly detection but does not work for data frame analytics.

and it says "an error occurred creating the data frame analytics job, Bad request [request.body.script\_fields] definition for this key is missing

Is it possible to use data frame analytics on a scripted field?

Thank you.

---

<div class="post-metadata">

**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\
**Post date:** [September 29, 2021, 7:08pm UTC](https://discuss.elastic.co/t/data-frame-analytics-on-scripted-fields-ml-possible/285504/2 "2021-09-29T19:08:44Z")

</div>

Great question @MLsuper ,

The answer is no. `script_fields` are not supported in Data frame analytics. But, since version `7.13.0` data frame analytics has supported `runtime_mappings` which can be used similarly to `script_fields`.

Example:

```auto
{
  "source": {
    "index": [
      "kibana_sample_data_flights" 
    ],
   "runtime_mappings": {
    "day_of_week": {
      "type": "keyword",
      "script": {
        "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"
      }
    }
    },
    "query": { 
      "range": {
        "DistanceKilometers": {
          "gt": 0
        }
      }
    },
    "_source": { 
      "includes": [],
      "excludes": [
        "FlightDelay",
        "FlightDelayType"
      ]
    }
  },
  "dest": { 
    "index": "df-flight-delays",
    "results_field": "ml-results"
  },
  "analysis": {
  "regression": {
    "dependent_variable": "FlightDelayMin",
    "training_percent": 90
    }
  },
  "analyzed_fields": { 
    "includes": [],
    "excludes": [
      "FlightNum"
    ]
  },
  "model_memory_limit": "100mb"
}

```

`day_of_week` is a categorical feature indicating the day of week of the flight, calculated at runtime from the `@timestamp` field via a script

---

<div class="post-metadata">

**Author:** ![MLsuper](https://avatars.discourse-cdn.com/v4/letter/m/dfb087/32.png) [@MLsuper](https://discuss.elastic.co/u/MLsuper)\
**Post date:** [September 29, 2021, 7:50pm UTC](https://discuss.elastic.co/t/data-frame-analytics-on-scripted-fields-ml-possible/285504/3 "2021-09-29T19:50:14Z")

</div>

I see, how is that runtime mapping used in this regression model? What does it do?

So currently I cannot use regression on my scripted fields?

Since it is not supported my plan is now to use Elasticsearch library in python and pull the data from ES-\>python and somehow manipulate the data and I can use a ML library(such as sklearn) and do regression that way?

Is it possible to get scripted field data from ES-\>python? Or will I have to do that on my own with python since scripted fields are only at run time?

EDIT:

right now my simple python code can grab data from ES using es.search, and getting the index pattern and playing around with it gives me the fields but not scripted fields. so if it is not possible I would have to loop all over the data and create the scripted field calculation for every point in python which seems like a difficult task?

thank you

---

<div class="post-metadata">

**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\
**Post date:** [September 29, 2021, 8:07pm UTC](https://discuss.elastic.co/t/data-frame-analytics-on-scripted-fields-ml-possible/285504/4 "2021-09-29T20:07:01Z")

</div>

@MLsuper

You can retrieve runtime\_mapping fields at query type by suppling the `runtime_mapping` in your Elasticsearch query, and then suppling the `fields` option including all the fields you want to retrieve (including runtime ones).

> **[Retrieve a runtime field | Elasticsearch Guide \[master\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/master/runtime-retrieving-fields.html)**

How do runtime\_mapping fields not satisfy your scripted field case? They can be used as features in data frame analytics and the scripts are simply `painless` scripts, just like scripted fields.

---

<div class="post-metadata">

**Author:** ![MLsuper](https://avatars.discourse-cdn.com/v4/letter/m/dfb087/32.png) [@MLsuper](https://discuss.elastic.co/u/MLsuper)\
**Post date:** [September 29, 2021, 8:10pm UTC](https://discuss.elastic.co/t/data-frame-analytics-on-scripted-fields-ml-possible/285504/5 "2021-09-29T20:10:04Z")

</div>

I see I will try this, and can I use this field as a dependent variable?

---

<div class="post-metadata">

**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\
**Post date:** [September 29, 2021, 8:25pm UTC](https://discuss.elastic.co/t/data-frame-analytics-on-scripted-fields-ml-possible/285504/6 "2021-09-29T20:25:20Z")

</div>

I don't see anything in the documentation that indicates if using it as the dependent variable is limited or not.

So, I am on the side that `runtime_mapping` fields should be able to be used as a dependent variable. 🙂

---

<div class="post-metadata">

**Author:** ![MLsuper](https://avatars.discourse-cdn.com/v4/letter/m/dfb087/32.png) [@MLsuper](https://discuss.elastic.co/u/MLsuper)\
**Post date:** [October 1, 2021, 3:45pm UTC](https://discuss.elastic.co/t/data-frame-analytics-on-scripted-fields-ml-possible/285504/7 "2021-10-01T15:45:31Z")

</div>

@BenTrent When I try to do the above it says

**An error occurred creating the data frame analytics job:**

**Bad Request: [request body.runtime\_mappings]: definition for this key is missing**

```auto
{
  "description": "",
  "source": {
    "index": "rotate-data"
  },
  
   "runtime_mappings": {
    "TimeDiff": {
      "type": "keyword",
      "script": {
        "source": "doc['@timestamp.max'].value.getMillis() - doc['@timestamp.min'].value.getMillis()"
      }
    }
    },

  "dest": {
    "index": "results-data"
  },
  "analyzed_fields": {
    "excludes": [

    ]
  },
  "analysis": {
    "regression": {
      "dependent_variable": "TimeDiff",
      "num_top_feature_importance_values": 2,
      "training_percent": 80
    }
  },
  "model_memory_limit": "73mb"
}

```

---

<div class="post-metadata">

**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\
**Post date:** [October 1, 2021, 6:53pm UTC](https://discuss.elastic.co/t/data-frame-analytics-on-scripted-fields-ml-possible/285504/8 "2021-10-01T18:53:14Z")

</div>

You must use an `emit` to expose the script value to the runtime field.

Also, for `regression` the dependent variable must be a numerical variable, so the output type should be a `long`

Also, `runtime_mappings` needs to be in the `source` definition object.

If this still doesn't work, please try via the Kibana Dev Console with the

```auto
PUT _ml/data_frame_analytics/my_analytics
{
  "description": "",
  "source": {
    "index": "rotate-data",
"runtime_mappings": {
    "TimeDiff": {
      "type": "long",
      "script": {
        "source": "emit(doc['@timestamp.max'].value.getMillis() - doc['@timestamp.min'].value.getMillis()_"
      }
    }
    }
  },
  "dest": {
    "index": "results-data"
  },
  "analyzed_fields": {
    "excludes": [
      "transaction_id.keyword",
      "transaction_id.keyword.keyword"
    ]
  },
  "analysis": {
    "regression": {
      "dependent_variable": "TimeDiff",
      "num_top_feature_importance_values": 2,
      "training_percent": 80
    }
  },
  "model_memory_limit": "73mb"
}

```

---

<div class="post-metadata">

**Author:** ![MLsuper](https://avatars.discourse-cdn.com/v4/letter/m/dfb087/32.png) [@MLsuper](https://discuss.elastic.co/u/MLsuper)\
**Post date:** [October 1, 2021, 7:28pm UTC](https://discuss.elastic.co/t/data-frame-analytics-on-scripted-fields-ml-possible/285504/9 "2021-10-01T19:28:06Z")

</div>

> [@BenTrent](#):
>
> `PUT _ml/data_frame_analytics/my_analytics`

I get this error

```auto
  "error" : {
    "root_cause" : [
      {
        "type" : "invalid_index_name_exception",
        "reason" : "Invalid index name [_ml], must not start with '_', '-', or '+'",
        "index_uuid" : "_na_",
        "index" : "_ml"
      }
    ],
    "type" : "invalid_index_name_exception",
    "reason" : "Invalid index name [_ml], must not start with '_', '-', or '+'",
    "index_uuid" : "_na_",
    "index" : "_ml"
  },
  "status" : 400
}

```

and if i remove the \_ in front of ml i get

```auto
{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "Rejecting mapping update to [ml] as the final mapping would have more than 1 type: [_doc, data_frame_analytics]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "Rejecting mapping update to [ml] as the final mapping would have more than 1 type: [_doc, data_frame_analytics]"
  },
  "status" : 400
}Preformatted text

```

when i try to put in kibana dev console since it does not work in data frame analytics same error as above

---

<div class="post-metadata">

**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\
**Post date:** [October 1, 2021, 7:41pm UTC](https://discuss.elastic.co/t/data-frame-analytics-on-scripted-fields-ml-possible/285504/10 "2021-10-01T19:41:56Z")

</div>

WOOPS! I did the wrong URL: [Create data frame analytics jobs API | Elasticsearch Guide [7.15] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-dfanalytics.html)

```auto
PUT _ml/data_frame/analytics/<data_frame_analytics_id>

```

---

<div class="post-metadata">

**Author:** ![MLsuper](https://avatars.discourse-cdn.com/v4/letter/m/dfb087/32.png) [@MLsuper](https://discuss.elastic.co/u/MLsuper)\
**Post date:** [October 1, 2021, 7:46pm UTC](https://discuss.elastic.co/t/data-frame-analytics-on-scripted-fields-ml-possible/285504/11 "2021-10-01T19:46:35Z")

</div>

Now I get this error

```auto
{
  "error" : {
    "root_cause" : [
      {
        "type" : "x_content_parse_exception",
        "reason" : "[5:1] [data_frame_analytics_source] unknown field [runtime_mappings]"
      }
    ],
    "type" : "x_content_parse_exception",
    "reason" : "[5:21] [data_frame_analytics_config] failed to parse field [source]",
    "caused_by" : {
      "type" : "x_content_parse_exception",
      "reason" : "[5:1] [data_frame_analytics_source] unknown field [runtime_mappings]"
    }
  },
  "status" : 400
}

```

I also tried instead of runtime mappings, I try replacing it with mappings and dynamic runtime like this:

```auto
PUT _ml/data_frame/analytics/<data_frame_analytics_id>
{
  "description": "",
  "source": {
    "index": "-qa",
"mappings": {
    "dynamic": "runtime",
    "TimeDiff": {
      "type": "long",
      "script": {
        "source": "emit(doc['@timestamp.max'].value.getMillis() - doc['@timestamp.min'].value.getMillis())"
      }
    }
    }
  },
  "dest": {
    "index": "results-regression"
  },
  "analyzed_fields": {
    "excludes": [
    ]
  },
  "analysis": {
    "regression": {
      "dependent_variable": "TimeDiff",
      "num_top_feature_importance_values": 2,
      "training_percent": 80
    }
  },
  "model_memory_limit": "73mb"
}

```

---

<div class="post-metadata">

**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\
**Post date:** [October 1, 2021, 8:01pm UTC](https://discuss.elastic.co/t/data-frame-analytics-on-scripted-fields-ml-possible/285504/12 "2021-10-01T20:01:13Z")

</div>

You have to have at least Elasticsearch version 7.13: [Create data frame analytics jobs API | Elasticsearch Guide [7.13] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.13/put-dfanalytics.html)

What version are you running?

---

<div class="post-metadata">

**Author:** ![MLsuper](https://avatars.discourse-cdn.com/v4/letter/m/dfb087/32.png) [@MLsuper](https://discuss.elastic.co/u/MLsuper)\
**Post date:** [October 1, 2021, 8:03pm UTC](https://discuss.elastic.co/t/data-frame-analytics-on-scripted-fields-ml-possible/285504/13 "2021-10-01T20:03:35Z")

</div>

Doing  
Get /  
gives me version 7.8.1

but minimum wire compatibility version is 6.8.0 and minimum index compatibility version is 6.0.0-beta1

---

<div class="post-metadata">

**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\
**Post date:** [October 1, 2021, 8:05pm UTC](https://discuss.elastic.co/t/data-frame-analytics-on-scripted-fields-ml-possible/285504/14 "2021-10-01T20:05:17Z")

</div>

Ah, well, the only way to get scripted fields for data frame analytics is to have version 7.13+ (and use runtime\_mappings).

---

<div class="post-metadata">

**Author:** ![MLsuper](https://avatars.discourse-cdn.com/v4/letter/m/dfb087/32.png) [@MLsuper](https://discuss.elastic.co/u/MLsuper)\
**Post date:** [October 1, 2021, 8:13pm UTC](https://discuss.elastic.co/t/data-frame-analytics-on-scripted-fields-ml-possible/285504/15 "2021-10-01T20:13:32Z")

</div>

Ahh, I see thank you, I will try this after updating

---

<div class="post-metadata">

**Author:** ![MLsuper](https://avatars.discourse-cdn.com/v4/letter/m/dfb087/32.png) [@MLsuper](https://discuss.elastic.co/u/MLsuper)\
**Post date:** [October 25, 2021, 4:38pm UTC](https://discuss.elastic.co/t/data-frame-analytics-on-scripted-fields-ml-possible/285504/16 "2021-10-25T16:38:58Z")

</div>

Hello Ben the cluster is upgraded and while doing regression I get this error, is it possible to do float in rune time mappings? for type:float? (it says invalid runtime mappings) because if i do long I get this error

cannot merge [properties] mappings because of differences for field [@timestamp]; mapped as [{properties={max={type=date}, min={type=date}, value\_count={type=long}}}] in index [p200]; mapped as [{properties={max={type=date}, min={type=date}, value\_count={type=float}}}] in index [p200-2021.10]

---

<div class="post-metadata">

**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\
**Post date:** [October 25, 2021, 4:51pm UTC](https://discuss.elastic.co/t/data-frame-analytics-on-scripted-fields-ml-possible/285504/17 "2021-10-25T16:51:44Z")

</div>

Yes, you can indicate `float` via your runtime mapping if you wish. The key is that the field type has to be the same for ALL indices matching your index pattern or it may error.

---

<div class="post-metadata">

**Author:** ![MLsuper](https://avatars.discourse-cdn.com/v4/letter/m/dfb087/32.png) [@MLsuper](https://discuss.elastic.co/u/MLsuper)\
**Post date:** [October 25, 2021, 5:17pm UTC](https://discuss.elastic.co/t/data-frame-analytics-on-scripted-fields-ml-possible/285504/18 "2021-10-25T17:17:23Z")

</div>

I see because when I try to enter "float" it doesn't let me click apply changes but if I change it to "double" or "long" it lets me click it.

cannot merge [properties] mappings because of differences for field [@timestamp]; mapped as [{properties={max={type=date}, min={type=date}, **value\_count={type=long** }}}] in index [p200]; mapped as [{properties={max={type=date}, min={type=date}, **value\_count={type=float** }}}] in index [p200-2021.10]

so this error the value\_count is **long** and is mapped to value\_count **float** but it has to be the same right[Screen Shot 2021-10-25 at 10.18.14 AM|690x182]

 ![Screen Shot 2021-10-25 at 10.19.29 AM](https://us1.discourse-cdn.com/elastic/original/3X/1/4/141c82ec09975ceb7b4c39bf076ef081c8b2fc08.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 22, 2021, 5:18pm UTC](https://discuss.elastic.co/t/data-frame-analytics-on-scripted-fields-ml-possible/285504/19 "2021-11-22T17:18:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.