I believe we are running the default sharding of 5 per index, across 7 index patterns, with each index-pattern generating a new index each day.
Question:
1). Is this sharding and index creation just plain wrong, given the use case (logging)
2). At what point do I require a dedicated master nodes? (at present they are just 5x data+master nodes)
Yes, that does not look good. You seen to have far too many indices and shards for a cluster that size. Please read this blog post on shards and sharding for practical guidance. I would recommend rethinking how you approach sharding and reduce this dramatically.
It often depends on the load of the system. If nodes are too busy and you start seeing long GC, master elections or cluster instability, it is probably time to add dedicated master nodes.
Also current document count for one index (as an example) is 17million
This will only grow perhaps by a factor of 10? as more systems are onboarded and send data...
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.