# Advanced ML job, transforming data

**URL:** <https://discuss.elastic.co/t/advanced-ml-job-transforming-data/142645>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-machine-learning\
**Created:** [August 1, 2018, 7:51pm UTC](https://discuss.elastic.co/t/advanced-ml-job-transforming-data/142645 "2018-08-01T19:51:55Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![palace](https://avatars.discourse-cdn.com/v4/letter/p/e47774/32.png) [@palace](https://discuss.elastic.co/u/palace)\
**Post date:** [August 1, 2018, 7:51pm UTC](https://discuss.elastic.co/t/advanced-ml-job-transforming-data/142645/1 "2018-08-01T19:51:55Z")

</div>

Hello,

I am making an advanced ML job in X-pack that will be partitioned by a certain job-type. At the moment, the data in the field job-type that is in ES includes the version number as: "job\_cats:release-4" (this is an example). If possible, I would like to trim all information after the colon for it to become "job\_cats" and partition my ML job on the job\_type without the version number (this is only useful for us to do for these ML jobs). Something similar to:

[https://www.elastic.co/guide/en/x-pack/current/ml-configuring-transform.html#ml-configuring-transform3](https://www.elastic.co/guide/en/x-pack/current/ml-configuring-transform.html#ml-configuring-transform3)

In short, how do I trim a string to include only information before the colon and then partition the field\_name with the new trimmed value.

Thank you in advanced

---

<div class="post-metadata">

**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\
**Post date:** [August 1, 2018, 8:55pm UTC](https://discuss.elastic.co/t/advanced-ml-job-transforming-data/142645/2 "2018-08-01T20:55:49Z")

</div>

You totally can. You will have to create the job via the API and partition on the to be created scripted field.

Then you can create a datafeed that creates the scripted field (may want to do a check with \_preview to verify your script is acting as it should).

And that should get you going. To my knowledge there is no way to do this strictly within the Kibana.

---

<div class="post-metadata">

**Author:** ![palace](https://avatars.discourse-cdn.com/v4/letter/p/e47774/32.png) [@palace](https://discuss.elastic.co/u/palace)\
**Post date:** [August 1, 2018, 9:16pm UTC](https://discuss.elastic.co/t/advanced-ml-job-transforming-data/142645/3 "2018-08-01T21:16:16Z")

</div>

So i can't do this in the "edit JSON" tab in the advanced ML jobs in Kibana?

And can you please provide an example of to trim my example with painless?

---

<div class="post-metadata">

**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\
**Post date:** [August 1, 2018, 9:26pm UTC](https://discuss.elastic.co/t/advanced-ml-job-transforming-data/142645/4 "2018-08-01T21:26:52Z")

</div>

You can create the Job in the Edit Json tab, but I am not 100% you can create the data feed there. You will have to create/update the datafeed outside of that.

As for the scripting part, something like the following SHOULD work, though I am no painless expert.

```
String[] parts = /:/.split(doc[‘job_type’].value); return parts[0];
```

---

<div class="post-metadata">

**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\
**Post date:** [August 2, 2018, 2:18pm UTC](https://discuss.elastic.co/t/advanced-ml-job-transforming-data/142645/5 "2018-08-02T14:18:27Z")

</div>

Example JSON Bodies

```
Job Creation 

{
"description" : "Unusual service behaviour",
"analysis_config" : {
    "bucket_span":"10m",
    "detectors" :[
      {
        "detector_description": "Unusual sum for each job",
        "function": "sum",
        "field_name": "total",
        "partition_field_name": "parsed_job_type"
      },
      {
        "detector_description": "Unusually high response for each job",
        "function": "high_mean",
        "field_name": "response",
        "partition_field_name": "parsed_job_type"
      }
    ],
    "influencers": ["host"]
    },
    "data_description" : {
      "time_field":"@timestamp",
      "time_format": "epoch_ms"
    }
}

Data-feed-creation

{
  "job_id" : "$JOB_ID",
  "indexes" : [
    "$INDEX_NAME"
  ],
  "types" : [
    "metric"
  ],
  "scroll_size" : 1000,
  "script_fields": {
     "parsed_job_type": {
        "script": {
          "lang": "painless",
          "source": "String[] parts = /:/.split(doc[‘job_type’].value); return parts[0];" 
        }
     }
  }
}
```

---

<div class="post-metadata">

**Author:** ![palace](https://avatars.discourse-cdn.com/v4/letter/p/e47774/32.png) [@palace](https://discuss.elastic.co/u/palace)\
**Post date:** [August 2, 2018, 4:16pm UTC](https://discuss.elastic.co/t/advanced-ml-job-transforming-data/142645/6 "2018-08-02T16:16:17Z")

</div>

Thank you so much, you're awesome!

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [October 29, 2018, 10:04pm UTC](https://discuss.elastic.co/t/advanced-ml-job-transforming-data/142645/7 "2018-10-29T22:04:21Z")

</div>