Hey guys, I've got a 6.5 PB cluster, composed of 150 d2.8xlarge instances (36 cores @ 2.5 GHz, 244 GB RAM, 24 hard disks (2 TB each, not in RAID)) that act as both data and ingest nodes. I have Filebeat running on 700+ prod service hosts, streaming around 3.5 GB/s of logs into hourly indexes. I've got a few questions regarding how I can optimize the performance of this cluster.
At the moment, I've configured my indexes to have 100 shards, and 0 replicas. Every hour, when the traffic starts pouring into a new index, I can see the CPU graphs of some servers drops, and other increase. This quite clearly shows which 100 servers were chosen for this hours' index, while the others are close to idle. Thus, I conclude that an a given server only contributes to indexing performance IF it stores a shard of the current index. Is this correct?
Secondly, from this page on maximizing index performance, I see that you guys recommend disabling replicas for initial loads of one-off data. That suggests to me that having 2 copies of a shard (1 primary, 1 replica) doubles that load, thus replication can't be used to increase indexing performance (although I understand that it does help with search performance).
So then, in order to get peak index performance out of my 150 nodes, I'll need an index with at least 150 shards. The issue there is that any one host going down will completely bring down the cluster. Are there any options? Have all hosts participate in the indexing, without each one becoming a risk to the whole cluster?
From what I gather, replica shards work like RAID 1, where you pay for 100% extra space, but have instant failover recovery. But there are many RAID schemes that can protect against a single drive failure, with only a small portion of extra space reserved for parity data. Recovering after a lost drive takes a little bit of processing time (to recompute the drive value from the other drives plus the parity), but it comes at the benefit of MUCH lower disk over-provisioning. Does Elasticsearch have anything similar?
When I get some free time, I'll work on switching to a hot/cold setup, but that poses its own challenges, because force-merging/shrinking on such large indexes becomes really difficult. Any advice?