Our free text search is unusable. We have a 3 master + 4 data/ingester setup.
We have 1 primary shard and 1 replica per index.
Indices are appended by date, and a new one is created every day (typical).
The daily indices are huge (>400GB).
Furthermore, the data plane resources are not being used efficiently. 2 of the data nodes are maxing out their requested (kubernetes) 3 CPU cores and utilizing much more disk space than 2 other data nodes that sit idle and are using less disk space.
What are we doing wrong? Should we increase replicas, primary shard count, both? Change our rollover strategy?