# Some help with write heavy ES cluster settings
**URL:** https://discuss.elastic.co/t/some-help-with-write-heavy-es-cluster-settings/40148
**Category:** Elasticsearch
**Created:** [January 26, 2016, 6:12pm UTC](https://discuss.elastic.co/t/some-help-with-write-heavy-es-cluster-settings/40148 "2016-01-26T18:12:43Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@dustin.liddick](https://discuss.elastic.co/u/dustin.liddick)
#### Post date: [January 26, 2016, 6:12pm UTC](https://discuss.elastic.co/t/some-help-with-write-heavy-es-cluster-settings/40148/1 "2016-01-26T18:12:43Z")
I am currently in the process of re-building my Elasticsearch cluster. I will have 4 data-nodes with 16GB of ram and 2vCPU and each data node will have a separate data disk that is 200GB in size. I will have 3 master-nodes, same CPU and RAM as the data nodes, but they will not have the additional disk added to them, and finally I will have 1 tribe node to handle the Kibana web front end calls and queries to elevate the load on the other nodes. I am hoping that this is proper.?
This (cluster/ELK) will be only for central logging of Cisco ASA logs, Red Hat syslogs, Windows tier 1 critical apps, and possibly some Palo Alto logs, but that may be added as a source later.
My question is that I would like some suggestions and pointers in configuring the ES cluster for this. I am assuming I am looking for "write heavy" settings, as the front end Kibana that will be the graphical end for other admins and non-techical people to view and aggregate the logs.
I will follow up with another comment to this thread with the config as it is the current config that I have come up with. This is from my automated install script, so its the data node setup script that this will be pasted from, if need be, I can past in the config I will be using for the master nodes, however, my understanding is that, the difference between the two is the 'node.master / node.data' call outs. I am hoping that I am on the right track, but also hoping others in the community might be setting something up similar to me. Appreciate and looking forward to the dialog.
---
### Author:  [@dustin.liddick](https://discuss.elastic.co/u/dustin.liddick)
#### Post date: [January 26, 2016, 6:13pm UTC](https://discuss.elastic.co/t/some-help-with-write-heavy-es-cluster-settings/40148/2 "2016-01-26T18:13:07Z")
############################### Logstash - Elasticsearch cluster Setup ##################################
# Register server with satellite
rhn-channel --add --channel=clone-epel\_rhel6x\_x86\_64 -u XXXX -p XXXX
# Install Oracle Java 8
echo "Installing Oracle Java 8"
mkdir /opt/collegis/software/java
cd /opt/collegis/software/java
wget --no-cookies --no-check-certificate --header "Cookie: gpw\_e24=http%3A%2F%[2Fwww.oracle.com](http://2Fwww.oracle.com)%2F; oraclelicense=accept-securebackup-cookie" "[http://download.oracle.com/otn-pub/java/jdk/8u20-b26/jdk-8u20-linux-x64.tar.gz](http://download.oracle.com/otn-pub/java/jdk/8u20-b26/jdk-8u20-linux-x64.tar.gz)"
tar -zxvf jdk-8u20-linux-x64.tar.gz
update-alternatives --install /usr/bin/java java /opt/collegis/software/java/jdk1.8.0\_20/bin/java 2
# Install Elasticsearch
rpm --import [https://packages.elastic.co/GPG-KEY-elasticsearch](https://packages.elastic.co/GPG-KEY-elasticsearch)
yum install -y --nogpgcheck elasticsearch
chkconfig --level 3 elasticsearch on
# Configuring Elasticsearch
echo "### Below is added using install script ###" \>\> /etc/elasticsearch/elasticsearch.yml
# Node name
echo "[cluster.name](http://cluster.name): collegis\_es\_cluster" \>\> /etc/elasticsearch/elasticsearch.yml
echo "[node.name](http://node.name): $yourhostname" \>\> /etc/elasticsearch/elasticsearch.yml
echo "node.datacenter: latisys" \>\> /etc/elasticsearch/elasticsearch.yml
echo "node.master: false" \>\> /etc/elasticsearch/elasticsearch.yml
echo "node.data: true" \>\> /etc/elasticsearch/elasticsearch.yml
echo "index.number\_of\_shards: 5" \>\> /etc/elasticsearch/elasticsearch.yml
echo "index.number\_of\_replicas: 1" \>\> /etc/elasticsearch/elasticsearch.yml
echo "bootstrap.mlockall: true" \>\> /etc/elasticsearch/elasticsearch.yml
echo "" \>\> /etc/elasticsearch/elasticsearch.yml
## Threadpool Settings
# Search pool
echo "threadpool.search.type: fixed" \>\> /etc/elasticsearch/elasticsearch.yml
echo "threadpool.search.size: 20" \>\> /etc/elasticsearch/elasticsearch.yml
echo "threadpool.search.queue\_size: 100" \>\> /etc/elasticsearch/elasticsearch.yml
echo "" \>\> /etc/elasticsearch/elasticsearch.yml
# Bulk pool
echo "threadpool.bulk.type: fixed" \>\> /etc/elasticsearch/elasticsearch.yml
echo "threadpool.bulk.size: 60" \>\> /etc/elasticsearch/elasticsearch.yml
echo "threadpool.bulk.queue\_size: 300" \>\> /etc/elasticsearch/elasticsearch.yml
echo "" \>\> /etc/elasticsearch/elasticsearch.yml
# Index pool
echo "threadpool.index.type: fixed" \>\> /etc/elasticsearch/elasticsearch.yml
echo "threadpool.index.size: 20" \>\> /etc/elasticsearch/elasticsearch.yml
echo "threadpool.index.queue\_size: 100" \>\> /etc/elasticsearch/elasticsearch.yml
echo "" \>\> /etc/elasticsearch/elasticsearch.yml
# Indices settings
echo "indices.memory.index\_buffer\_size: 30%" \>\> /etc/elasticsearch/elasticsearch.yml
echo "indices.memory.min\_shard\_index\_buffer\_size: 12mb" \>\> /etc/elasticsearch/elasticsearch.yml
echo "indices.memory.min\_index\_buffer\_size: 96mb" \>\> /etc/elasticsearch/elasticsearch.yml
echo "" \>\> /etc/elasticsearch/elasticsearch.yml
# Cache Sizes
echo "indices.fielddata.cache.size: 15%" \>\> /etc/elasticsearch/elasticsearch.yml
echo "indices.fielddata.cache.expire: 6h" \>\> /etc/elasticsearch/elasticsearch.yml
echo "indices.cache.filter.size: 15%" \>\> /etc/elasticsearch/elasticsearch.yml
echo "indices.cache.filter.expire: 6h" \>\> /etc/elasticsearch/elasticsearch.yml
echo "" \>\> /etc/elasticsearch/elasticsearch.yml
# Indexing Settings for Writes
echo "index.refresh\_interval: 30s" \>\> /etc/elasticsearch/elasticsearch.yml
echo "index.translog.flush\_threshold\_ops: 50000" \>\> /etc/elasticsearch/elasticsearch.yml
echo "" \>\> /etc/elasticsearch/elasticsearch.yml
# Minimum nodes alive to constitute an operational cluster
echo "#### Prevent split brain ES Cluster n/2+1 ####" \>\> /etc/elasticsearch/elasticsearch.yml
echo "discovery.zen.minimum\_master\_nodes: 2" \>\> /etc/elasticsearch/elasticsearch.yml
echo "#" \>\> /etc/elasticsearch/elasticsearch.yml
echo 'discovery.zen.ping.unicast.hosts: ["nodename-1p", "nodename-2p", "nodename-3p", "nodename-4p", "nodename-5p", "nodename-6p", "nodename-7p", "nodename-8p"]' \>\> /etc/elasticsearch/elasticsearch.yml
echo "#discovery.zen.ping.multicast.enabled: false" \>\> /etc/elasticsearch/elasticsearch.yml
# Making changes to /etc/security/limits.conf to allow more open files for elasticsearch
mv /etc/security/limits.conf /etc/security/limits.bak
grep -Ev "# End of file" /etc/security/limits.bak \> /etc/security/limits.conf
echo "elasticsearch soft nofile 65536" \>\> /etc/security/limits.conf
echo "elasticsearch hard nofile 65536" \>\> /etc/security/limits.conf
echo "elasticsearch - memlock unlimited" \>\> /etc/security/limits.conf
echo "# End of file" \>\> /etc/security/limits.conf
# Modify elasticsearch service for ulimit -l unlimited to allow mlockall to work correctly
sed -i -e 's|^#ES\_HEAP\_SIZE=2g|ES\_HEAP\_SIZE=16g|' /etc/init.d/elasticsearch
sed -i -e 's|^#MAX\_LOCKED\_MEMORY=|MAX\_LOCKED\_MEMORY=unlimited|' /etc/init.d/elasticsearch
# Set Elasticsearch to start on boot
chkconfig elasticsearch on
# Set Elasticsearch to start on boot
chkconfig elasticsearch on
# Restart Elasticsearch service
service elasticsearch restart
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [January 26, 2016, 6:53pm UTC](https://discuss.elastic.co/t/some-help-with-write-heavy-es-cluster-settings/40148/3 "2016-01-26T18:53:49Z")
Do not play with search pool, bulk pool, index pool, indices settings, cache size, indexing settings for writes (except temporary change to refresh\_interval while bulk indexing is active), unless you have to.
Just leave the defaults. ES 2.1+ will autobalance all this for you. Note, many tunables will be gone or become unmodifiable in the future, for instance, thread pool sizes.
---
### Author:  [@dustin.liddick](https://discuss.elastic.co/u/dustin.liddick)
#### Post date: [January 26, 2016, 7:26pm UTC](https://discuss.elastic.co/t/some-help-with-write-heavy-es-cluster-settings/40148/4 "2016-01-26T19:26:57Z")
jprante:
Much appreciated for that advice. Since I had been running on 1.4, and now it will be 2.0 this is great news. Ill run with defaults for now and configure by hand each one, not that big of an issue for me!!
Again, thanks for that information and tips
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 11:21pm UTC](https://discuss.elastic.co/t/some-help-with-write-heavy-es-cluster-settings/40148/5 "2017-07-05T23:21:18Z")