How to create and use data for indexing using Rally

Elastic Stack Elasticsearch
1

We are using Rally to do performance benchmarking on our ELK stacks. For this we want to use our own data (like apm events) to ingest to elastic and measure index performance.

https://github.com/elastic/apm-server/tree/master/rally has the dataset for apm but looks like it is old data (2 years old) and is not working for current versions of ES 7xx.

How can we create our own data, compress it and use for indexing with Rally? Is there some documentation or guidelines listed that can help in this.

Please share.

Thanks

2

Hello,

Thank you for your interest in Rally.

Regarding creating your own custom track please take a look at: https://esrally.readthedocs.io/en/stable/adding_tracks.html#example-track

Regards,
Dimitris

3

Thank you but my query is regarding creation of dataset. How can i create dataset with my own indexes data like for apm events.
Any pointers for that?

Thanks

4

There are a few enhancement requests in this area e.g. making it easier for Rally to use existing data for benchmarks and to be able to generate load based on the slowlog (which can log most traffic if threshold is set low enough). Please add comments and a +1 if any of these approaches would work for you.

As it stands now I do not think there is any official way to generate test data and track configuration for existing indices.

5

This PR in particular is about a tool that will help generating a track from an existing ES cluster.

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.