ELK architecture advice with S3

On the masters front: the answer is always "3 dedicated master nodes" and anything else is an architectural compromise. The rationale for this is :

Dedicated because asking a master node to also perform indexing or searching duties can put it under stress. A master node must perform an important (but lightweight) managerial function so shouldn't be stressed by data loads in the same way you do with data nodes or clusters can become unresponsive
3 because 1 represents a single-point-of-failure and 2 has the potential for split-brain.

Of course not everyone builds a cluster using 3 dedicated master nodes but it is important to know the trade-offs you're making when you stray from this advice.