Sharding of APM indexes

Elastic Observability APM
1

Need suggestion on sharding for APM indexes. Our span index(per day) will consume 7-8GB of data. The default number of shards per index is 1.

I want to know is it recommended to have a single shard per index (based on the in-built apm dashboard queries and indexing performance. Also the index size is not too big in our case) or should I increase the shard count as per our requirements?

Thanks,
Shashank

2

Increasing the number of shards helps to horizontally scale reads and writes. However, each shard comes with a certain overhead. In fact, the ideal shard size is around 25GB-50GB.

What are your requirements and scalability needs? How many Elasticsearch nodes do you have? Do you feel like the default is not working for you?

What you could also do is to leverage curator or index lifecycle management to have a higher amount of shards for the current index and to shrink to one shard for older indices.

1 Like
3

Thanks Felix for the response.
Given below is our cluster information:
3 node ES cluster
64GB ram
250GB SSD
number of replicas: 1
APM indexes to be stored for last 30 days

We will mainly use the built-in APM dashboard for monitoring our services. Max query per second will be around 70.

About the default configuration, I feel my cluster will be under utilized given that I have 3 nodes.

Using curator or index lifecycle management sounds good for our use case. Will work on that.

4

I wouldn't worry about increasing shard count in this case. You can increase replicas to improve read performance (aka search), but unless you're seeing noticeable slowness it's going to just waste heap.

5

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.