# Benchmarking es-cluster on AWS

**URL:** <https://discuss.elastic.co/t/benchmarking-es-cluster-on-aws/307636>\
**Category:** Elasticsearch\
**Tags:** rally\
**Created:** [June 20, 2022, 11:01am UTC](https://discuss.elastic.co/t/benchmarking-es-cluster-on-aws/307636 "2022-06-20T11:01:29Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![amitsa](https://avatars.discourse-cdn.com/v4/letter/a/dbc845/32.png) [@amitsa](https://discuss.elastic.co/u/amitsa)\
**Post date:** [June 20, 2022, 11:01am UTC](https://discuss.elastic.co/t/benchmarking-es-cluster-on-aws/307636/1 "2022-06-20T11:01:29Z")

</div>

Hi All,

I am running my es-cluster on kubernetes on AWS instance x2iedn(memory optimized) and i4i(Storage optimized).  
I tried benchmarking the es cluster on both instance type.

I have 3 data nodes and 1 master node.  
each have a heap size 31 gb.

Resources of container is requested as below.

```auto
resources:
            requests:
              cpu: 8
              memory: "64Gi"

```

I am using nyc\_taxis track to benchmark cluster.  
I see the throughput for index follows as  
32x instance \< 16x instance \< 8x instance \> 4x instance

i.e suppose

4x ---\> 4000 doc/s  
8x ----\> 4300 docs/sec  
16x ---\> 3900 doc/sec  
32x ---\> 3800 doc/s

these are just sample not exact throughput.

I have shards: 20  
replicas: 0  
bulksize: 10000  
client : 8  
refresh\_interval: -1

The throughput should increase as we go for higher capacity instance.

But i see it starts reducing from 16x instance and reduces further for 32x instance(less than 4x instance).

Kindly help me with this.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 20, 2022, 11:14am UTC](https://discuss.elastic.co/t/benchmarking-es-cluster-on-aws/307636/2 "2022-06-20T11:14:41Z")

</div>

When it comes to indexing performance, the amount of memory available is less important than the performance of the storage. The instance type you are using seem better suited to high concurrent query loads as all data might be able to fit in the operating system page cache. What type of storage are you using with the instance?

You should also note that the standard tracks are not necessarily set up to load very powerful nodes by default and you may very well need to tweak settings and concurrentcy in order to make Rally generate enough load to saturate the cluster, maybe even run multiple Rally instances.

I am also curious what you are looking to get out of this exercise. Does the expected workload for the cluster at all resemble the track you are using? I always recommend creating a custom track that as realistically as possible represents the data and load you are expecting in the cluster and then use this to get an as accurate estimate as possible.

I would also recommend the following resources:

> **[Using Rally to Get Your Elasticsearch Cluster Size Right](https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right)**

> **[Seven Tips for Better Elasticsearch Benchmarks](https://www.elastic.co/blog/seven-tips-for-better-elasticsearch-benchmarks)**

---

<div class="post-metadata">

**Author:** ![amitsa](https://avatars.discourse-cdn.com/v4/letter/a/dbc845/32.png) [@amitsa](https://discuss.elastic.co/u/amitsa)\
**Post date:** [June 20, 2022, 11:24am UTC](https://discuss.elastic.co/t/benchmarking-es-cluster-on-aws/307636/3 "2022-06-20T11:24:20Z")

</div>

Hi @Christian_Dahlqvist

Thankyou for the response.

I am just looking to achieve the maximum indexing throughput by utilizing the available resources to the maximum.

I am just looking into any sample data which i can use to saturate my cluster to get maximum throughput.

Also my allocated resources gets used to maximum so that i can gain maximum throuhput index.

Kindly suggest what should i do to acheive it.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 20, 2022, 12:01pm UTC](https://discuss.elastic.co/t/benchmarking-es-cluster-on-aws/307636/4 "2022-06-20T12:01:22Z")

</div>

I would recommend having a look at the resources I linked to.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 18, 2022, 12:01pm UTC](https://discuss.elastic.co/t/benchmarking-es-cluster-on-aws/307636/5 "2022-07-18T12:01:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.