I've been reading alot on how many shards / replicas I need, and I'm seeing multiple opinions.
We have a 3 node cluster that mainly receives logs through logstash from multiple sources. We have around 30 different types going into 15 or so different indicies. Logstash breaks these indicies into daily indicies, which results in around 320 total indicies. Most all of them were setup from the start with 5 shards and 1 replica. This results in 2,800 shards. We have around 270 million documents that results in 590 GB of data on 3 nodes. Our cluster is constantly receiving data from log sources but doesn't receive a large amount of queries: around 12 users look at the data sparingly throughout the day.
What would be the best shard / replica setup for this type of cluster? Can I reduce the shard sizes per index to 1 per node and 1 replica (3 shards and 1 replica)? Is there a way to keep the queries fast but reduce the over data stored on disk?
Also, what is the best way to test performance if I do change shard /replicas ? How do I tell if queries are taking longer of performance is suffering?