# Questions about custom tracks

**URL:** <https://discuss.elastic.co/t/questions-about-custom-tracks/118794>\
**Category:** Elasticsearch\
**Tags:** rally\
**Created:** [February 7, 2018, 10:00am UTC](https://discuss.elastic.co/t/questions-about-custom-tracks/118794 "2018-02-07T10:00:36Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\
**Post date:** [February 7, 2018, 10:00am UTC](https://discuss.elastic.co/t/questions-about-custom-tracks/118794/1 "2018-02-07T10:00:36Z")

</div>

Hi,

last time I tried to upgrade my productive ES instance from 5.1.2 to 6.1.0 I got performance issues afterwards.

Before running the next rollout, I would like to do some loadtests before to be sure that changes during my migration / integration do nothing bad.

I got the hint to use rally and here I am.  
I would like to test as near to reality as possible by spending as few time as possible for that 😉

We are indexing log files, splitting it to fields and we have multiple extensive dashboards which are visualized via kibana.

Is the assumption correct, that the kind of data (number of indexes, number of docs per index, number of shards, number of fields per index, number of fields per type and of course the user queries (via kibana)) are (significantly) effecting the speed of my node / cluster?

If so, I assume I need to build my own track.

- is it possible to export productive data and use it as document input for rally? How?
- How do I deal with the user queries? Is there an easy way to take the queries e.g. from kibana logs and provide them as challenge? How?

In the meantime I will continue reading the docs (I am not completely through yet) and will get warm with the default tracks.

Thanks, Andreas

---

<div class="post-metadata">

**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\
**Post date:** [February 7, 2018, 12:00pm UTC](https://discuss.elastic.co/t/questions-about-custom-tracks/118794/2 "2018-02-07T12:00:20Z")

</div>

> [@asp](#):
>
> We are indexing log files, splitting it to fields and we have multiple extensive dashboards which are visualized via kibana.

Sounds like the [eventdata track](https://github.com/elastic/rally-eventdata-track) could be a good starting point for you. Note that this track generates data on the fly though (normally we bulk-index an already existing data set).

> [@asp](#):
>
> Is the assumption correct, that the kind of data (number of indexes, number of docs per index, number of shards, number of fields per index, number of fields per type and of course the user queries (via kibana)) are (significantly) effecting the speed of my node / cluster?

Yes.

> [@asp](#):
>
> If so, I assume I need to build my own track.

The closer your model is to your production workload the more useful are the numbers that you get out of it. So I'd agree here.

> [@asp](#):
>
> is it possible to export productive data and use it as document input for rally? How?

Rally needs a line-delimited file that is in a [bulk-friendly format](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html). There is no tooling in Rally but can use a small script to get data out of Elasticsearch and write it into a line-delimited JSON file. You could base your script on something like [CLI for elaasticsearch-py helpers · GitHub](https://gist.github.com/HonzaKral/0fb3f76da529ee380d08).

> [@asp](#):
>
> How do I deal with the user queries? Is there an easy way to take the queries e.g. from kibana logs and provide them as challenge? How?

Not at the moment but we plan to add some support for that soon. The idea is that you use Elasticsearch's slow log to generate the query workload, for details see [#262](https://github.com/elastic/rally/issues/262) on Github.

I'd choose the queries that are most important to you, determine your current workload (e.g. by turning the slowlog theshold to zero for a little while) and then model the number of clients and the target throughput accordingly. [Modelling client arrivals](http://esrally.readthedocs.io/en/stable/track.html#choosing-a-schedule) gets quite important if you want to simulate several clients. For independent clients, Poisson-based arrival are a good choice (see the docs for details).

You might also want to watch:

- [Using Rally to Get Your Elasticsearch Cluster Size Right](https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right)
- [Quantitative Cluster Sizing](https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing)

(both are free to watch but require prior registration).

At this year's ElasticON, I'll also give a talk about benchmarking pitfalls, called [The Seven Deadly Sins of Elasticsearch Benchmarking](https://www.elastic.co/elasticon/conf/2018/sf/agenda?sess=the-seven-deadly-sins-of-elasticsearch-benchmarking). I don't know for sure but I'd expect that the contents are available a little while after the conference (as it was the case in previous years).

---

<div class="post-metadata">

**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\
**Post date:** [February 7, 2018, 1:39pm UTC](https://discuss.elastic.co/t/questions-about-custom-tracks/118794/3 "2018-02-07T13:39:03Z")

</div>

Thanks for the answer. Also the 1st mentioned video was very interesting. Looks like the part I need, but with our queries and data.

Another idea came into my mind about how creating the input json. I could use logstash and output the data as json lines to a file. Should be compatible, right?

When I use a file as input, is it also possible throttle the ingestion speed? So that I could set the throughput like on production?

Is it possible to import from multiple files simulaneously to different indices / types?

Then hopefully I could replay the productive load.

I will read through the event-data-track and hope to understand what Christian is doing there. (I am not familar with python yet ☹ )

---

<div class="post-metadata">

**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\
**Post date:** [February 7, 2018, 2:18pm UTC](https://discuss.elastic.co/t/questions-about-custom-tracks/118794/4 "2018-02-07T14:18:42Z")

</div>

> [@asp](#):
>
> Another idea came into my mind about how creating the input json. I could use logstash and output the data as json lines to a file. Should be compatible, right?

I would think so. As long as you have one line per document you should be good. You can decide to include the action and meta-data line in that file directly or have Rally generate it for you on the fly (see the [track reference for `includes-action-and-meta-data`](http://esrally.readthedocs.io/en/stable/track.html#corpora)).

> [@asp](#):
>
> When I use a file as input, is it also possible throttle the ingestion speed? So that I could set the throughput like on production?

Yes, you can set [`target-throughput`](http://esrally.readthedocs.io/en/stable/track.html#schedule) to throttle it. One caveat: The target throughput is operation-agnostic (i.e. it does not know about bulks) and is measured in requests per second (not documents per second).

> [@asp](#):
>
> Is it possible to import from multiple files simulaneously to different indices / types?

Yes. By default, Rally will just ingest all data files that you provide it (one after the other). If you want to have more control over that process then you can leverage the [`parallel`](http://esrally.readthedocs.io/en/stable/track.html#running-tasks-in-parallel) element but then you need explicitly specify an index list for your [`bulk` operation](http://esrally.readthedocs.io/en/stable/track.html#bulk). Otherwise Rally would ingest all your indices twice (remember: ingesting everything is the default behavior). I'm thinking something along the lines of:

```auto
{
  "schedule": [
    {
      "parallel": {
        "tasks": [
          {
            "operation": {
              "operation-type": "bulk",
              "bulk-size": 5000,
              "indices": ["logs"]
            },
            "warmup-time-period": 120,
            "clients": 8,
            "target-throughput": 1
          },
          {
            "operation": {
              "operation-type": "bulk",
              "bulk-size": 2000,
              "indices": ["users"]
            },
            "warmup-time-period": 120,
            "clients": 4,
            "target-throughput": 2
          }
        ]
      }
    }
  ]
}

```

(beware: totally untested)

> [@asp](#):
>
> I will read through the event-data-track and hope to understand what Christian is doing there. (I am not familar with python yet ☹ )

This track is quite advanced and Christian pushed Rally quite to the edge 🙂

If you have static data, then you can probably just model it as a regular track and don't need any Python at all. One more tip: I'd stick to the examples in the docs and not base your track on the [standard Rally tracks](https://github.com/elastic/rally-tracks) because they implement a few workarounds so users with older versions of Rally can still run them. That makes them a bit harder to understand than the examples in the docs that only target a specific Rally version.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 7, 2018, 2:19pm UTC](https://discuss.elastic.co/t/questions-about-custom-tracks/118794/5 "2018-03-07T14:19:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.