Some help with write heavy ES cluster settings

dustin.liddick · January 26, 2016, 6:12pm

I am currently in the process of re-building my Elasticsearch cluster. I will have 4 data-nodes with 16GB of ram and 2vCPU and each data node will have a separate data disk that is 200GB in size. I will have 3 master-nodes, same CPU and RAM as the data nodes, but they will not have the additional disk added to them, and finally I will have 1 tribe node to handle the Kibana web front end calls and queries to elevate the load on the other nodes. I am hoping that this is proper.?

This (cluster/ELK) will be only for central logging of Cisco ASA logs, Red Hat syslogs, Windows tier 1 critical apps, and possibly some Palo Alto logs, but that may be added as a source later.

My question is that I would like some suggestions and pointers in configuring the ES cluster for this. I am assuming I am looking for "write heavy" settings, as the front end Kibana that will be the graphical end for other admins and non-techical people to view and aggregate the logs.

I will follow up with another comment to this thread with the config as it is the current config that I have come up with. This is from my automated install script, so its the data node setup script that this will be pasted from, if need be, I can past in the config I will be using for the master nodes, however, my understanding is that, the difference between the two is the 'node.master / node.data' call outs. I am hoping that I am on the right track, but also hoping others in the community might be setting something up similar to me. Appreciate and looking forward to the dialog.

dustin.liddick · January 26, 2016, 6:13pm

############################### Logstash - Elasticsearch cluster Setup ##################################

Register server with satellite

rhn-channel --add --channel=clone-epel_rhel6x_x86_64 -u XXXX -p XXXX

Install Oracle Java 8

echo "Installing Oracle Java 8"
mkdir /opt/collegis/software/java
cd /opt/collegis/software/java
wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u20-b26/jdk-8u20-linux-x64.tar.gz"
tar -zxvf jdk-8u20-linux-x64.tar.gz
update-alternatives --install /usr/bin/java java /opt/collegis/software/java/jdk1.8.0_20/bin/java 2

Install Elasticsearch

rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
yum install -y --nogpgcheck elasticsearch
chkconfig --level 3 elasticsearch on

Configuring Elasticsearch

echo "### Below is added using install script ###" >> /etc/elasticsearch/elasticsearch.yml

Node name

echo "cluster.name: collegis_es_cluster" >> /etc/elasticsearch/elasticsearch.yml
echo "node.name: $yourhostname" >> /etc/elasticsearch/elasticsearch.yml
echo "node.datacenter: latisys" >> /etc/elasticsearch/elasticsearch.yml
echo "node.master: false" >> /etc/elasticsearch/elasticsearch.yml
echo "node.data: true" >> /etc/elasticsearch/elasticsearch.yml
echo "index.number_of_shards: 5" >> /etc/elasticsearch/elasticsearch.yml
echo "index.number_of_replicas: 1" >> /etc/elasticsearch/elasticsearch.yml
echo "bootstrap.mlockall: true" >> /etc/elasticsearch/elasticsearch.yml
echo "" >> /etc/elasticsearch/elasticsearch.yml

Threadpool Settings

Search pool

echo "threadpool.search.type: fixed" >> /etc/elasticsearch/elasticsearch.yml
echo "threadpool.search.size: 20" >> /etc/elasticsearch/elasticsearch.yml
echo "threadpool.search.queue_size: 100" >> /etc/elasticsearch/elasticsearch.yml
echo "" >> /etc/elasticsearch/elasticsearch.yml

Bulk pool

echo "threadpool.bulk.type: fixed" >> /etc/elasticsearch/elasticsearch.yml
echo "threadpool.bulk.size: 60" >> /etc/elasticsearch/elasticsearch.yml
echo "threadpool.bulk.queue_size: 300" >> /etc/elasticsearch/elasticsearch.yml
echo "" >> /etc/elasticsearch/elasticsearch.yml

Index pool

echo "threadpool.index.type: fixed" >> /etc/elasticsearch/elasticsearch.yml
echo "threadpool.index.size: 20" >> /etc/elasticsearch/elasticsearch.yml
echo "threadpool.index.queue_size: 100" >> /etc/elasticsearch/elasticsearch.yml
echo "" >> /etc/elasticsearch/elasticsearch.yml

Indices settings

echo "indices.memory.index_buffer_size: 30%" >> /etc/elasticsearch/elasticsearch.yml
echo "indices.memory.min_shard_index_buffer_size: 12mb" >> /etc/elasticsearch/elasticsearch.yml
echo "indices.memory.min_index_buffer_size: 96mb" >> /etc/elasticsearch/elasticsearch.yml
echo "" >> /etc/elasticsearch/elasticsearch.yml

Cache Sizes

echo "indices.fielddata.cache.size: 15%" >> /etc/elasticsearch/elasticsearch.yml
echo "indices.fielddata.cache.expire: 6h" >> /etc/elasticsearch/elasticsearch.yml
echo "indices.cache.filter.size: 15%" >> /etc/elasticsearch/elasticsearch.yml
echo "indices.cache.filter.expire: 6h" >> /etc/elasticsearch/elasticsearch.yml
echo "" >> /etc/elasticsearch/elasticsearch.yml

Indexing Settings for Writes

echo "index.refresh_interval: 30s" >> /etc/elasticsearch/elasticsearch.yml
echo "index.translog.flush_threshold_ops: 50000" >> /etc/elasticsearch/elasticsearch.yml
echo "" >> /etc/elasticsearch/elasticsearch.yml

Minimum nodes alive to constitute an operational cluster

echo "#### Prevent split brain ES Cluster n/2+1 ####" >> /etc/elasticsearch/elasticsearch.yml
echo "discovery.zen.minimum_master_nodes: 2" >> /etc/elasticsearch/elasticsearch.yml
echo "#" >> /etc/elasticsearch/elasticsearch.yml
echo 'discovery.zen.ping.unicast.hosts: ["nodename-1p", "nodename-2p", "nodename-3p", "nodename-4p", "nodename-5p", "nodename-6p", "nodename-7p", "nodename-8p"]' >> /etc/elasticsearch/elasticsearch.yml
echo "#discovery.zen.ping.multicast.enabled: false" >> /etc/elasticsearch/elasticsearch.yml

Making changes to /etc/security/limits.conf to allow more open files for elasticsearch

mv /etc/security/limits.conf /etc/security/limits.bak
grep -Ev "# End of file" /etc/security/limits.bak > /etc/security/limits.conf
echo "elasticsearch soft nofile 65536" >> /etc/security/limits.conf
echo "elasticsearch hard nofile 65536" >> /etc/security/limits.conf
echo "elasticsearch - memlock unlimited" >> /etc/security/limits.conf
echo "# End of file" >> /etc/security/limits.conf

Modify elasticsearch service for ulimit -l unlimited to allow mlockall to work correctly

Set Elasticsearch to start on boot

chkconfig elasticsearch on

Set Elasticsearch to start on boot

chkconfig elasticsearch on

Restart Elasticsearch service

service elasticsearch restart

jprante · January 26, 2016, 6:53pm

Do not play with search pool, bulk pool, index pool, indices settings, cache size, indexing settings for writes (except temporary change to refresh_interval while bulk indexing is active), unless you have to.

Just leave the defaults. ES 2.1+ will autobalance all this for you. Note, many tunables will be gone or become unmodifiable in the future, for instance, thread pool sizes.

dustin.liddick · January 26, 2016, 7:26pm

jprante:

Much appreciated for that advice. Since I had been running on 1.4, and now it will be 2.0 this is great news. Ill run with defaults for now and configure by hand each one, not that big of an issue for me!!

Again, thanks for that information and tips