# Possible reason for \> 30% difference between iterations of geonames -\> painless\_static **URL:** \ **Category:** Elasticsearch\ **Tags:** rally\ **Created:** [March 29, 2019, 8:22am UTC](https://discuss.elastic.co/t/possible-reason-for-30-difference-between-iterations-of-geonames-painless-static/174493 "2019-03-29T08:22:21Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\ **Post date:** [March 29, 2019, 8:22am UTC](https://discuss.elastic.co/t/possible-reason-for-30-difference-between-iterations-of-geonames-painless-static/174493/1 "2019-03-29T08:22:21Z")
During my task to compare performance between dockerized elasticsearch and running elasticsearch as native service / on shell, I discovered big fluctuation between the tests **- when rerunning the same testset!** First I compared the overall time of the run: ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/e/0ef93190d6d24caf66bdadba614403ecf0f07338.png) so we van see about 4 minutes difference in overall time. I digged deeper to see which operation is causing this: ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/2/829f1dd6990e55bc87292815b7d49b6185a5b80d.png) There are a few operations which are slower, but painless\_static is the negative winner. ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/3/b3e00afa24c6d2ec7e79476786f8f0eaecbf4696.png) Above we see throughput, latency and service time. The second run is much slower. But I am not able to find anything suspcios in metricbeat dashboards: ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/6/9667f33885020615b6c1be395e5375a8a17aa32a.png) see 12:24 - 12:28 and compare with 13:02 - 13:09 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/3/63817863fe890e5a5823b367e12b5e6f95a99d00.png) Here is my testing process: - delete old geonames-index via kibana - stop elasticsearch - stop kibana - start elasticsearch - start kibana - wait until elasticsearch is up - run esrally with external car, track geonames, challenge default. Nothing else is running on the server where elasticsearch is tested (except for metricbeat). esrally runs on a system shared with elastic dev system with very low load. But I also stopped that dev system in previous runs, where the benchmarks had the same fluctuation. Any help is really appreciated. Thanks a lot, Andreas ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [April 1, 2019, 2:21pm UTC](https://discuss.elastic.co/t/possible-reason-for-30-difference-between-iterations-of-geonames-painless-static/174493/2 "2019-04-01T14:21:16Z")
Hi Andreas, to understand what's happening I suggest to watch my talk [The Seven Deadly Sins of Elasticsearch Benchmarking](https://www.elastic.co/elasticon/conf/2018/sf/the-seven-deadly-sins-of-elasticsearch-benchmarking) (free to watch but requires prior registration). Please check item "sin 3" which covers your question extensively. See also the related blog post [Seven Tips for Better Elasticsearch Benchmarks](https://www.elastic.co/blog/seven-tips-for-better-elasticsearch-benchmarks) which is a summary of the talk. Daniel ---
**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\ **Post date:** [April 2, 2019, 6:10am UTC](https://discuss.elastic.co/t/possible-reason-for-30-difference-between-iterations-of-geonames-painless-static/174493/3 "2019-04-02T06:10:28Z")
Hey Daniel, Thanks for the reply. your workshop was interesting. It's clear to me that the latency will go up, if we query faster than the system can respond because auf the raising queue. What is not completely clear to me is that, why the service time is varying that much. Do you think it is caused by the overload? And if I lower the target throughput the values should become more stable? Regards, Andreas ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [April 9, 2019, 12:05pm UTC](https://discuss.elastic.co/t/possible-reason-for-30-difference-between-iterations-of-geonames-painless-static/174493/4 "2019-04-09T12:05:04Z")
> [@asp](#): > > What is not completely clear to me is that, why the service time is varying that much. The talk (and in fact Rally as well) is making a simplifying assumption, namely that the benchmarked system can be modelled using only one queue (also known as [M/M/1 queue](https://en.wikipedia.org/wiki/M/M/1_queue) in queuing theory) but systems in practice can have several queues, e.g. incoming network packets can queue up on OS level, runnable processes queue up in the CPU scheduler's run queue, Elasticsearch has a queue in front of its thread pool and if multiple Elasticsearch nodes are processing a query even more queues are involved. So service time is only an approximation (although the best one that we can get from a client perspective) and that would explain why you see a varying service time. > [@asp](#): > > Do you think it is caused by the overload? And if I lower the target throughput the values should become more stable? As a corollary from my previous reasoning, this could indeed be the case and would make sense to test. Daniel ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 7, 2019, 12:17pm UTC](https://discuss.elastic.co/t/possible-reason-for-30-difference-between-iterations-of-geonames-painless-static/174493/5 "2019-05-07T12:17:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.