We receive approx. 1TB of data everyday(including replica) for 100+ indexes. We currently have 16TB storage spread which stores data for 15 days. Out ES cluster has 2 dedicates MI nodes and 6 Di nodes and 1 MDI node. (M = master D = data I = Ingest)
We would like to optimize our cluster for better storage (of 30 days) and index/search performance.
Should we have dedicated master nodes(with no ingestion)?
Should we add client nodes for query purposes?
Can we use best_compression for storage? how would it affect my search queries?
We create a snapshot of the current index state every day. In case we plan to restore index for any day which is around 150 Gb, it takes approx 2 hours to restore. Is there a faster way to restore indexes from the snapshot?
How many indices/shards do you have in your cluster? Based on your description it sounds like you may be having a lot of small indices and shards, which can be very inefficient.
Yes. Allowing the dedicated master nodes to just manage the cluster is recommended as best practice as it improves cluster stability. You should also make sure you have 3 of them.
There is no clear recommendation on this. It depends a lot on your use case.
It generally does not affect searching, but will require a bit extra work at indexing time.
The defaults for restoring snapshots are set to not overwhelm the cluster, but can be tweaked. I believe the restore process uses the shard recovery mechanism, so you can increase the indices.recovery.max_bytes_per_sec setting to make it more aggressive. You should also be able to alter the shard allocation settings.
We have over 100 micro-services, so each micro-service has its own index. We have default shard number set to 6. Our index size differs from few MBs to 500-660 GB.
Also currently some of our nodes in the cluster have primary shards only. Is there a setting which needs to be tuned so that all primary shards in the cluster are equally distributed on nodes and so are the replicas after that.
That sounds quite inefficient. I would recommend changing the shard count according to the index sizes to better align with the recommendations in the blog post I linked to. If you primarily are indexing new data and not updating documents, primaries and replicas basically do the same work, so an uneven distribution of primary shards may not matter much.
But we have observed that:
As the indexes are time-series, all of them get created at 12 AM UTC. The problem is that all the primary shards of most of the indexes are allocated to one node(say N1) and few primary shards and replicas are assigned to other nodes.
Cluster configuration set to:
cluster.routing.rebalance.enable: all
cluster.routing.allocation.allow_rebalance: indices_all_active
thread_pool.bulk.queue_size: 1000
During bulk indexing requests, all of the requests go to that node (N1) and CPU utilization increases for this node. A lot of requests are also rejected as the queue size exceeds on that node. Whereas other nodes stay chilled out.
Doubts:
Is the above issue, is it because all the primary shards are on one node only?
If yes, Can I rebalance my primary shards by setting "cluster.routing.rebalance.enable" to "primaries". Would this configuration first rebalance my primary shards and then balance the replicas? Are there any repercussions.
Is there any other cause of the issue and is there a way to mitigate it?
As far as I know there is no way to efficiently control this as Elasticsearch need to manage this in response to index and cluster events.
As outlined in this blog post, writing to lots of shards can be inefficient and quickly fill up bulk indexing queues. Ways to improve this would be to dramatically reduce the number of shards being written to and/or try to ensure each bulk request target a smaller set of indices/shards, e.g. through the use of multiple discreet pipelines and pipeline-to-pipeline communication.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.