Various questions about an Elastic cluster

Elastic Stack Elasticsearch
1

I have successfully set up an Elastic cluster with X-Pack and various Beats to simulate a production setting. However, I encountered some issues/questions while I was playing with the cluster.

My setup is like this:

  • One master node running Ubuntu 16.04 (10.0.2.4) running Elasticsearch (to do query and stuff), Logstash, Kibana.
  • Two nodes (10.0.2.5 - Ubuntu 16.04, 10.0.2.15 - CentOS 7) running Elasticsearch (to store replicas and shards) and all the Beats that deliver data to Logstash and Elasticsearch on 10.0.2.4.

My /etc/hosts file for all 3 machines contains

10.0.2.4 elastic-master.server
10.0.2.5 elastic-ubuntu.server
10.0.2.15 elastic-centos.server

On all 3 machines, elasticsearch.yml has this line

discovery.zen.ping.unicast.hosts: ["elastic-master.server", "elastic-ubuntu.server", "elastic-centos.server"]

  1. Suppose I want to add a new node to the cluster. Do I edit all existing Elasticsearch nodes in the cluster to add the new node? What if the cluster already has a lot of nodes, say, 50 nodes. Do I have to edit all 50 nodes?

  2. I tried to simulate a DDoS by sending 10000 logs/s from Filebeats on elastic-ubuntu and elastic-centos to Logstash on elastic-master. Some shards quickly died after that due to lack of free space. When I checked, there were about 3 million logs taking 6GB. What would be a good minimum amount of space for a production environment? Let's say I run a small website that receives several hundred to a thousand views/day and I want to monitor the traffic (This is just an example for me to get a feel of the resources needed).

  3. The index rate was painfully low, maxing around 1k2/s. What should I do to increase it?

2

1 master is bad, see https://www.elastic.co/guide/en/elasticsearch/guide/2.x/important-configuration-changes.html#_minimum_master_nodes

  1. You just need to list the master (eligible) nodes.
  2. Depends, what sort of data is it, what is your mapping? What's your expected EPS?
  3. What sort of hardware? What did you give ES for heap? What about Logstash?
3
  1. Just normal Apache logs. I'm following the example in https://github.com/elastic/examples/tree/master/Common Data Formats/apache_logs
  2. VirtualBox VM with 8GB RAM assigned. I haven't changed any other ES and Logstash settings.
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.