Installation questions

Elastic Stack Elasticsearch
1

Hello, all.

Apologies if this is the wrong place to post these questions, or if they're laughably stupid, but I need your help. Up to now, I've been running a single-node, version 2.x, ELK stack. While we have only about 100 servers shipping logs to it, it is under a bit of strain. So, now I figure it's time set up a multi-node, version 5.x, ELK stack. My intention is to do that from scratch, rather than upgrade my current instance. The following questions (and there will probably be more)) are important to my effort, as I can't seem to find the answers anywhere else (though they may be out there). Here goes:

  1. What is the optimal, or perhaps minimum, number of nodes to include in my cluster (from what I've read, it seems to be three)?
  2. What roles should the new nodes be set up for (Master, DAta, etc.)?
  3. Which ELK applications live on which nodes?
  4. How would config files for Logstash, Elasticsearch, etc. differ between nodes?
  5. To which node(s) do the various servers ship their logs to?

Hope I'm not being a pita but, again, I really need the help to move forward.

Diggy

2

Hi Diggy,

here's how I have my elastic stack set up.

I have two physical machines which each run 5 Elasticsearch nodes as I have 5 data drives, one dedicated to each ES node. It doesn't really matter if you run separate machines for each node, which I think is the preferred approach but I couldn't choose my hardware so for me it made sense to run more than one node per machine.

I have one Logstash instance and one Kibana instance per physical machine.

The machines have 126 GB RAM and 24 CPU threads. Each ES node have a 10 GB heap. I think Logstash has the same, which leaves about half of the RAM for Lucene/system.

All nodes are data nodes. 4 of them are also master eligible. I'm using discovery.zen.minimum_master_nodes: 3. The reason 3 master eligible nodes are the minimum for high availability is because you need quorum, a majority of master nodes that decide what is the correct current state of the cluster. With 3 master nodes that quorum is 2 which translates to discovery.zen.minimum_master_nodes: 2

For a 3 node cluster I would run all nodes as master, data and ingestion (that the new one, right) nodes. If the machines have enough hardware resources I would also run Logstash and Kibana on each.

If you use Filebeat as the logshipper you can set all Logstash instances to receive logs and Filebeat will loadbalance between them.

For the config files per node, I think the only difference would be network.host. If you try to run more instances of ES on the same machine then there's more differences.

Good luck :slight_smile:

3

Thanks for the response, A_B. Other responses still encouraged and welcomed.

I use filebeat, syslog on a few CentOS 5 boxes (which are in the process of being upgraded), nxlog, winlogbeat, and topbeat. Do all logstash instances be set up to receive these logs, as well?

Diggy

4

Again, just from my own experience and setup. We have the same Logstash instances for all types of logs.

Input types are lumberjack, beats, syslog and UDP.

5

Thanks, again, A_B. I would appreciate any additional responses so that I make sure I get this right. Anyone else please feel free.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.