This is a x-post from Slack, but I think that it is perhaps easier to have a discussion here.
We are setting up a new logcluster that should handle .. logs.. from our infrastructure mostly. The usual suspects filebeat, winlogbeat, auditbeat etc.
Our design after some testing in docker landed on this.
But after a brief discussion in Slack I was recommended not to use "Master" role on the Ingest nodes (internal client facing).
Is this design sane, for a 14 days of hot storage and then a few months of Cold storage for "archive"?
Looking at this, a question I would have is for your ingest does that include, Elasticsearch ingest role nodes? If it does, one thing you'll need to considered is I don't believe Elasticsearch allows for layer 3 clustering, so the subnet for all servers, must be the same (I bring this up as I'm assuming mgmt-pub and mgmt-sec are different subnets). Elasticsearch Ingest nodes are becoming increasingly important with filebeat modules.
Oh, I haven't seen that before. I thought that the cluster "talked" unicast with the other cluster nodes?
Do you have a link to the docs with this info, so I can read up about it?
After looking a bit deeper, it appears that you're correct, in that if you're using unicast discovery, then you can span multiple subnets.
What I was thinking of was multi-region which is not recommended, but also not your use case. So it doesn't apply.
One last thing then, I think you'll need port 9300/tcp open between mgmt-pub and mgmt-sec so that the Elasticsearch ingest role nodes can properly talk to the rest of the ES nodes.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
