Hello everybody,
first post here in the community.
I setup an ES 5.4.0 cluster on AWS made of 5 data-nodes and 3 masters (only), to collect all the logs from our application.
The data nodes are pretty OK servers (8CPU, 32GiB), 1 SSD volume of 2TB for data nodes, and 100GB for masters, all EBS optimized.
Heap set to 16GB, and I basically followed what the guide says about setting it up.
I have a daily index rotation, with 6 indexes per day.
Default template is set with 2 shards and 2 replicas, whereby 3 of them have 5 shards and 3 replicas (but thinking about having 1 replica only to save space, once cluster is stable).
best-compression is enabled on all indexes.
I am using td-agent (fluentd) to ship logs directly to ES; no parsing is needed as they are already in JSON format.
I started with dynamic mapping but for the biggest indexes (read below) I will move to manual mapping to save space, and hopefully gain a bit on speed.
Problem is, those 3 indexes are quite big, and the indexing rate is quite high.
One index is already at production level, and those are the numbers:
8 docker instances writing at ~2000/s (primary shard) and 30GB data so far (I am looking at today's index via Kibana monitoring); refresh_interval at 15s, dynamic mapping off and _all field off.
The other 2 indexes are as follows:
1 server out of 8:
- first index at 1200/s rate (primary shard) with 60m documents and 35GB space;
- second index is at 500/s with 20m docs but 450GB data (unfortunately those are quite big log files)
refresh_interval for the above 2 indexes is at 30s, dynamic mapping still on.
Problem is, if I enable fluentd on all 8 servers, the data nodes can't cope with the amount of incoming logs.
Few things I can (and should) optimize:
- disable dynamic mapping on the latter indexes, and disable _all field to save some space;
- disable best-compression (I read it consumes processing power, so better to set it off and enable it for yesterday's indexes while running force_merge)
Current conf on elasticsearch.yml:
# Get pool
thread_pool.get.queue_size: 200
# Index pool
thread_pool.index.queue_size: 600
# Search pool
thread_pool.search.queue_size: 200
# Bulk pool
thread_pool.bulk.queue_size: 1000
Currently, for JVM I see the following values (read it from one of the data nodes):
GC old is 0, young at around 2500ms avg
GC count old 0, young at around 125
Indexing time avg 200000ms (4 minutes)
I know the above conf does not really help as it hides the true root cause, but reading here and there I found I was getting rejected documents so I increased the bulk.queue _size and lowered the searches (we don't do much searches on it).
I read the part about tune for indexing speed and one thing I didn't understand was the "indices.memory.index_buffer_size" value.
Where should I set this and most importantly how can I see the current value?
Most likely it needs to be set on a per-index basis meaning I would have to update my templates including that parameter.
Is there any other tips/values you can suggest me to look after and adjust based on the incoming flow?
The cluster can keep up with 2 of 8 servers sending data, but I tried setting 4 of them and I see load on the data nodes going above 8.00 avg (which should be their max given they are 8-core servers).
I would like to understand where the bottleneck of my cluster is; if it's related to some setting I forgot to configure or simply there is too much data needed to be indexed and I should scale-out the data nodes.
For the sake of completeness, at first I setup a similar cluster using AWS ES and Kinesis Firehose to send data, but I had problems with space as the could have only 1 disk per server with maximum of 1.5TB space; plus it was too expensive to keep it running so I decided to go with EC2 instances and a load balancer on top to connect only the master nodes (as I later understood there is no need to have all the nodes connected to the load balancer, only the masters are enough).
Thank you in advance,
Lorenzo
