# Using esrally to measure painless script partial update performance

**URL:** <https://discuss.elastic.co/t/using-esrally-to-measure-painless-script-partial-update-performance/262263>\
**Category:** Elasticsearch\
**Tags:** rally\
**Created:** [January 26, 2021, 5:37pm UTC](https://discuss.elastic.co/t/using-esrally-to-measure-painless-script-partial-update-performance/262263 "2021-01-26T17:37:34Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![dineshabbi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dineshabbi/32/56301_2.png) [@dineshabbi](https://discuss.elastic.co/u/dineshabbi)\
**Post date:** [January 26, 2021, 5:37pm UTC](https://discuss.elastic.co/t/using-esrally-to-measure-painless-script-partial-update-performance/262263/1 "2021-01-26T17:37:34Z")

</div>

Hi team,

I am trying to measure bulk partial update performance. I am wondering if its possible to do this via esrally at all.

For eg, our application uses painless scripts for partial update and the target for these updates are individual docs since application is aware of \_id:

```auto
POST index-001/doc/{_id=BlahBlah}
{
// body of partial update message
}

```

To update a document via esrally, I may have to use update\_by\_query API

```auto
POST index-001/_update_by_query
  "query": {
    "primaryId": "BlahBlah"
  },
  "script": {
    "source": "abs(ln(abs(doc['population']) + 1) + doc['location'].lon + doc['location'].lat) * _score",
    "lang": "painless"
  }

```

IIRC esrally runs against index rather than against individual docs, hence need to rely on update\_by\_query. This index/update\_by\_query is acceptable for me and would provide comparable results. In my experiment, I am loading the original docs first and then I run esrally with corpus of incremental updates which are driven through stored\_scripts (have a matching index field between original and incremental doc).

However, esrally seems to support only painless script "search" operation out of the box ☹. I couldn't find anything for update or bulk update.

```auto
      "operation-type" : "search",
      """body": {
        "query": {
          "function_score": {
            "query": {
              "primaryId": "BlahBlah"
            },
            "functions": [
              {
                "script_score": {
                  "script": {
                    "source": "abs(ln(abs(doc['population']) + 1) + doc['location'].lon + doc['location'].lat) * _score",
                    "lang": "painless"
                  }
                }

```

Please lemm eknow if there are any other ideas. I am not sure if custom runners would solve our problem of non-existant operation-type, and any pointers to custom runner example may help.

> **[Define Custom Workloads: Tracks — Rally 2.0.3 documentation](https://esrally.readthedocs.io/en/stable/adding_tracks.html#custom-runners)**

  

> **[Track Reference — Rally 2.0.3 documentation](https://esrally.readthedocs.io/en/stable/track.html#operations)**

  

> **[Track Reference — Rally 2.0.3 documentation](https://esrally.readthedocs.io/en/stable/track.html#anatomy-of-a-track)**

Tx,

---

<div class="post-metadata">

**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\
**Post date:** [January 27, 2021, 8:15am UTC](https://discuss.elastic.co/t/using-esrally-to-measure-painless-script-partial-update-performance/262263/2 "2021-01-27T08:15:34Z")

</div>

Hi @dineshabbi,

Thanks for your interest in Rally.

If I understood correctly, you'd like to use `update_by_query` which is currently not supported as a native operation in Rally.

You can easily create a [custom runner](https://esrally.readthedocs.io/en/stable/adding_tracks.html?highlight=runner#custom-runners), to build this new operation. You'd directly use [update\_by\_query](https://elasticsearch-py.readthedocs.io/en/v7.10.1/async.html?highlight=update_by_query#elasticsearch.AsyncElasticsearch.update_by_query) method from the Elasticsearch Python client. I haven't tested it, but I believe that you'd just specify your query and script in the `body` as shown e.g. in the [Elasticsearch example](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html#docs-update-by-query-api-source).

Finally if you don't want to use a custom runner (and/or if a dedicated API call is missing from the elasticsearch-py client) you can always use the [raw-request](https://esrally.readthedocs.i/en/stable/track.html?highlight=raw-request#raw-request) operation to invoke any ES Rest API.

---

<div class="post-metadata">

**Author:** ![dineshabbi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dineshabbi/32/56301_2.png) [@dineshabbi](https://discuss.elastic.co/u/dineshabbi)\
**Post date:** [January 27, 2021, 11:25am UTC](https://discuss.elastic.co/t/using-esrally-to-measure-painless-script-partial-update-performance/262263/3 "2021-01-27T11:25:34Z")

</div>

Thanks @dliappis for great pointers ! One follow-up Qn: currently our application performs bulk painless updates. I was curious if both the above approaches you suggested let me measure the performance of bulk updates via esrally. I guess if I figure out a way to POST bulk http REST call via raw-request or custom runner approach, this may work, but not very sure.

We are not planning to implement custom runner unless there are no other alternatives for us. raw-request may help us as long as it lets us meeasure bulk partial updates too.

---

<div class="post-metadata">

**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\
**Post date:** [January 28, 2021, 12:38pm UTC](https://discuss.elastic.co/t/using-esrally-to-measure-painless-script-partial-update-performance/262263/4 "2021-01-28T12:38:31Z")

</div>

`raw-request` will contain the usual metrics and you'll have the individual samples for `service_time`, `latency`, `error_rate` and `throughput` (in `rally-results` as well as individual metric records in the index `rally-metrics`) .

With a custom runner in addition to those you have the chance to enhance results with other metrics of your choice, see for example [this example](https://github.com/elastic/rally-eventdata-track/blob/686c959c6d6c127ced48ae6a4edc738cf32cadbb/eventdata/runners/kibana_runner.py#L91-L108).

It's worth reading [https://esrally.readthedocs.io/en/stable/recipes.html?highlight=result#checking-queries-and-responses](https://esrally.readthedocs.io/en/stable/recipes.html?highlight=result#checking-queries-and-responses) as well.

---

<div class="post-metadata">

**Author:** ![dineshabbi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dineshabbi/32/56301_2.png) [@dineshabbi](https://discuss.elastic.co/u/dineshabbi)\
**Post date:** [February 3, 2021, 7:27am UTC](https://discuss.elastic.co/t/using-esrally-to-measure-painless-script-partial-update-performance/262263/5 "2021-02-03T07:27:43Z")

</div>

Thanks again @dliappis .

1. With bulk partial update, what I was looking for was something like this:

```auto
POST _bulk
{ "update" : { "_id" : "ID1", "_index" : "t1_item", "retry_on_conflict" : 3} }
{ "script" : { "id": "item-update-script", "params" : {"partialUpdate":{"supplierOfTradeItem":[{"primaryId": "_primaryIdBulk2_","additionalPartyId": [{"value": "_value_","typeCode": "FOR_BULK_USE_1"}],"name": "_name_","isPrimarySupplier": false,"avpList": [{"value": "value","name": "name_bulk2","actionCode":"ADD"},{"value": "value","name": "name_bulk3","actionCode":"ADD"}]}]}}}}

{ "update" : { "_id" : "ID2", "_index" : "t1_item", "retry_on_conflict" : 3} }
{ "script" : { "id": "item-update-script", "params" : {"partialUpdate":{"supplierOfTradeItem":[{"primaryId": "_primaryIdBulk2_","additionalPartyId": [{"value": "_value_","typeCode": "FOR_BULK_USE_1"}],"name": "_name_","isPrimarySupplier": false,"avpList": [{"value": "value","name": "name_bulk2","actionCode":"ADD"},{"value": "value","name": "name_bulk3","actionCode":"ADD"}]}]}}}}

```

I am curious if the custom runner allows me to inject this API. I will explore that path since it becomes tedious to supply the whole body with raw-request for bulk.

1. Couple years ago, I remember esrally used to store the benchmark standard metrics index /rally-metrics-\*/ in ES cluster itself.  
[Metrics — Rally 2.0.3 documentation](https://esrally.readthedocs.io/en/stable/metrics.html)  
However, I am no longer seeing that default behavior. How can I store the metrics on a dedicated ES index in the cluster ?

---

<div class="post-metadata">

**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\
**Post date:** [February 3, 2021, 9:07am UTC](https://discuss.elastic.co/t/using-esrally-to-measure-painless-script-partial-update-performance/262263/6 "2021-02-03T09:07:46Z")

</div>

> [@dineshabbi](#):
>
> Few years ago, I remember esrally used to store the benchmark staandard metrics in one of the indices in ES cluster itself. However, I am no longer seeing that default behavior. How can I store the every run metrics on a dedicated ES index in the cluster ?

Yes of course, this is the preferred way of using Rally, as having the metrics in an Elasticsearch cluster gives the possibility to explore data with Kibana visualizations. See the [docs here](https://esrally.readthedocs.io/en/stable/configuration.html?highlight=metrics%20store#reporting).

Also since you mentioned:

> store the benchmark staandard metrics in one of the indices in ES cluster itself

I wanted to highlight that storing benchmark metrics in the **same cluster** you are benchmarking is an **anti-pattern**. The ES cluster you are benchmarking should be doing just that, i.e. receiving only benchmark related role, not experience load from other activities like storing metrics. Instead, your metrics store should be a different Elasticsearch cluster (it doesn't need to be highly available, or very powerful/large).

> [@dineshabbi](#):
>
> 1. With bulk partial update, what I was looking for was something like this:
> 
> `POST _bulk`

This is different to what we were [discussing earlier](https://discuss.elastic.co/t/using-esrally-to-measure-painless-script-partial-update-performance/262263) which was the `_update_by_query` API.

Given that there is already a [bulk operation](https://esrally.readthedocs.io/en/stable/track.html#bulk) in Rally and that you can specify [action-and-metadata](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk-api-desc) in the [corpora section](https://esrally.readthedocs.io/en/stable/track.html#corpora) of your track using the `include-action-and-metadata` property, you could simply use your example above as your corpora.

I came up with the following example:

I have an existing Elasticsearch cluster containing docs like:

```auto
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "logs-181998",
        "_type" : "_doc",
        "_id" : "oR7zZncBQLUmKToTl7Tr",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : 893999489,
          "clientip" : "253.27.0.0",
          "request" : "GET /images/hm_anime_e.gif HTTP/1.0",
          "status" : 200,
          "size" : 15609
        }
      },
      {
        "_index" : "logs-181998",
        "_type" : "_doc",
        "_id" : "Wh7zZncBQLUmKToTl5Xe",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : 894196437,
          "clientip" : "39.164.0.0",
          "request" : "GET /english/images/news_btn_kits_off.gif HTTP/1.1",
          "status" : 200,
          "size" : 933
        }
      }
    ]
  }
}

```

Then I created the following simple Rally track `updatescripttrack` to update these two docs to show status `404` instead of `200`:

The corpora:

```auto
me@server ~/updatescripttrack $ cat documents.json 
{"update": {"_id": "oR7zZncBQLUmKToTl7Tr", "_index" : "logs-181998" } }
{"script": "ctx._source.status = 404" }
{"update": {"_id": "Wh7zZncBQLUmKToTl5Xe", "_index" : "logs-181998" } }
{"script": "ctx._source.status = 404" }

```

The track itself:

```auto
me@server ~/updatescripttrack $ cat track.json 
{
  "version": 2,
  "description": "Test Rally with update script workload",
  "indices": [
    {
      "name": "logs-181998"
    }
  ],
  "corpora": [
    {
      "name": "testupdate",
      "documents": [
        {
          "source-file": "documents.json",
          "includes-action-and-meta-data": true,
          "document-count": 2
        }
      ]
    }
  ],
  "schedule": [
    {
      "operation": {
        "operation-type": "bulk",
        "bulk-size": 1
      }
    }
  ]
}

```

Which I then ran using:

`esrally --pipeline=benchmark-only --track-path=~/updatescripttrack --on-error=abort`

which successfully did the change:

```auto
GET logs-181998/_search
{
  "query": {
    "terms": {
      "_id": ["oR7zZncBQLUmKToTl7Tr", "Wh7zZncBQLUmKToTl5Xe"]
    }
  }
}

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "logs-181998",
        "_type" : "_doc",
        "_id" : "oR7zZncBQLUmKToTl7Tr",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : 893999489,
          "clientip" : "253.27.0.0",
          "request" : "GET /images/hm_anime_e.gif HTTP/1.0",
          "status" : 404,
          "size" : 15609
        }
      },
      {
        "_index" : "logs-181998",
        "_type" : "_doc",
        "_id" : "Wh7zZncBQLUmKToTl5Xe",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : 894196437,
          "clientip" : "39.164.0.0",
          "request" : "GET /english/images/news_btn_kits_off.gif HTTP/1.1",
          "status" : 404,
          "size" : 933
        }
      }
    ]
  }
}

```

Rally will report the standard bulk operation metrics as with any other bulk operation.

---

<div class="post-metadata">

**Author:** ![dineshabbi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dineshabbi/32/56301_2.png) [@dineshabbi](https://discuss.elastic.co/u/dineshabbi)\
**Post date:** [February 3, 2021, 9:42am UTC](https://discuss.elastic.co/t/using-esrally-to-measure-painless-script-partial-update-performance/262263/7 "2021-02-03T09:42:02Z")

</div>

> [@dliappis](#):
>
> storing benchmark metrics in the **same cluster** you are benchmarking is an **anti-pattern**

Agree. I will check the cost incurred on this. I am not running anything other than update, hence hoping that the metrics shouldn't skew my results way too much. If so, will move them to another host.

> [@dliappis](#):
>
> This is different to what we were [discussing earlier] which was the `_update_by_query` API.

Correct, I later realized that I didn't mean to ask the "bulk" notion of update by update\_by\_query, rather my bulk operation meant for bundling multiple update requests at client and send them at once.

> [@dliappis](#):
>
> Given that there is already a [bulk operation](https://esrally.readthedocs.io/en/stable/track.html#bulk) in Rally and that you can specify [action-and-metadata](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html#docs-bulk-api-desc) in the [corpora section](https://esrally.readthedocs.io/en/stable/track.html#corpora) of your track using the `include-action-and-metadata` property, you could simply use your example above as your corpora.

Awesome !! This is exactly what I was looking for. Thank you so much for this pointer.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 3, 2021, 9:42am UTC](https://discuss.elastic.co/t/using-esrally-to-measure-painless-script-partial-update-performance/262263/8 "2021-03-03T09:42:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.