# Machine Learning on scripted field?

**URL:** <https://discuss.elastic.co/t/machine-learning-on-scripted-field/89054>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-machine-learning\
**Created:** [June 12, 2017, 1:56pm UTC](https://discuss.elastic.co/t/machine-learning-on-scripted-field/89054 "2017-06-12T13:56:26Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![bcz](https://avatars.discourse-cdn.com/v4/letter/b/b5e925/32.png) [@bcz](https://discuss.elastic.co/u/bcz)\
**Post date:** [June 12, 2017, 1:56pm UTC](https://discuss.elastic.co/t/machine-learning-on-scripted-field/89054/1 "2017-06-12T13:56:26Z")

</div>

I want to be able to run machine learning on an ingested field, but need to do a transformation to that field to make it useable. The transformation is the conversion of a score from a log scale to a linear scale.

I have set up a scripted field in Kibana called Linear\_Score and I can see the calculated values in Discover. But when I try to set up a single metric job on the field, no preview of scores is shown. When I run the job, it spins for a minute, then stops after looking through 0 documents. There is no error message or anything else.

After some digging, this article

> **[Using Painless in Kibana scripted fields
	  	 | Elastic](https://www.elastic.co/blog/using-painless-kibana-scripted-fields)**
>
> Kibana provides powerful ways to search and visualize data stored in Elasticsearch. For the purpose of visualizations, Kibana looks for fields defined in Elasticsearch mappings and presents them as op...

  
makes me think that I might need to make this script part of the elastic query itself. But, I can't figure out where in the json query I need to put the scripted field so it will be part of the learning.

I can't find any documentation on this and there are no helpful error messages. I can write my own scripted field in Dev Tools, but nothing is working when I add the code to the machine learning job JSON.

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [June 12, 2017, 3:52pm UTC](https://discuss.elastic.co/t/machine-learning-on-scripted-field/89054/2 "2017-06-12T15:52:41Z")

</div>

Hello,

Yes, creating a scripted\_field must be part of the datafeed configuration ([https://www.elastic.co/guide/en/x-pack/5.4/ml-put-datafeed.html](https://www.elastic.co/guide/en/x-pack/5.4/ml-put-datafeed.html)).

To accomplish:

1. Create an advanced job (only can do this technique on advanced jobs)
2. Configure the job as you normally would. When creating detectors, instead of choosing an existing field, choose a name that we'll later assign to the script field. In this example, we are choosing the field name total\_error\_count which doesn't exist in our documents
3. Once your job is configured as you like, go to the "Edit JSON" tab
4. Append a new script\_fields parameter inside the datafeed\_config object. The syntax for script\_fields is identical to that used by Elasticsearch. You can find [more information on the syntax here](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-script-fields.html). We'll add our total\_error\_count script field to the script\_fields object. The script will do a simple addition of two fields in the document to produce a "total" error count:

```auto
  "datafeed_config": {
    "query": {
      "match_all": {}
    },
    "query_delay": "60s",
    "frequency": "150s",
    "scroll_size": 1000,
    "indexes": [
      "rally-2017"
    ],
    "types": [
      "metrics"
    ],
    "script_fields": {
      "total_error_count": {
        "script": {
          "lang": "painless",
          "inline": "doc['error_count'].value + doc['aborted_count'].value"
        }
      }
    }
  }

```

When done editing the JSON, you can verify the output of your script with the results with the "Data Preview" tab. When satisfied, press Save.

You'll notice that our detector referenced "total\_error\_count", which is generated at runtime by the script. Every time a document is loaded by Elasticsearch, the script is evaluated and its result outputted as a "virtual" field. This is then used by the ML job.

---

<div class="post-metadata">

**Author:** ![bcz](https://avatars.discourse-cdn.com/v4/letter/b/b5e925/32.png) [@bcz](https://discuss.elastic.co/u/bcz)\
**Post date:** [June 12, 2017, 5:18pm UTC](https://discuss.elastic.co/t/machine-learning-on-scripted-field/89054/3 "2017-06-12T17:18:17Z")

</div>

Thank you! That is working now and I also hadn't come across the documentation you pointed me towards.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 10, 2017, 5:18pm UTC](https://discuss.elastic.co/t/machine-learning-on-scripted-field/89054/4 "2017-07-10T17:18:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.