I have setup a cluster of 3 ES nodes (5.1.1), all three nodes are listed in the elasticsearch output plugin in logstash and the cluster is healthy.
Looking at the thread_pool, one of my nodes does not seem to be doing as much as the others and it is not a master.
Every now and again it processes one bulk request but most of the time its only one management thread active.
ES master/data
ES1 data
ES2 data
node_name name active queue rejected
elk-poc-es2 bulk 4 10 14
elk-poc-es2 fetch_shard_started 0 0 0
elk-poc-es2 fetch_shard_store 0 0 0
elk-poc-es2 flush 0 0 0
elk-poc-es2 force_merge 0 0 0
elk-poc-es2 generic 0 0 0
elk-poc-es2 get 0 0 0
elk-poc-es2 index 0 0 0
elk-poc-es2 listener 0 0 0
elk-poc-es2 management 1 0 0
elk-poc-es2 refresh 0 0 0
elk-poc-es2 search 0 0 0
elk-poc-es2 snapshot 0 0 0
elk-poc-es2 warmer 0 0 0
elk-poc-es bulk 4 49 0
elk-poc-es fetch_shard_started 0 0 0
elk-poc-es fetch_shard_store 0 0 0
elk-poc-es flush 0 0 0
elk-poc-es force_merge 0 0 0
elk-poc-es generic 0 0 0
elk-poc-es get 0 0 0
elk-poc-es index 0 0 0
elk-poc-es listener 0 0 0
elk-poc-es management 1 0 0
elk-poc-es refresh 0 0 0
elk-poc-es search 0 0 0
elk-poc-es snapshot 0 0 0
elk-poc-es warmer 0 0 0
elk-poc-es1 bulk 1 0 0
elk-poc-es1 fetch_shard_started 0 0 0
elk-poc-es1 fetch_shard_store 0 0 0
elk-poc-es1 flush 0 0 0
elk-poc-es1 force_merge 0 0 0
elk-poc-es1 generic 0 0 0
elk-poc-es1 get 0 0 0
elk-poc-es1 index 0 0 0
elk-poc-es1 listener 0 0 0
elk-poc-es1 management 1 0 0
elk-poc-es1 refresh 1 0 0
elk-poc-es1 search 0 0 0
elk-poc-es1 snapshot 0 0 0
elk-poc-es1 warmer 0 0 0
It also seems most under utilised
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.0.0.1 58 98 46 3.58 3.48 3.19 mdi - elk-poc-es2
10.0.0.2 17 98 74 5.19 4.68 4.75 mdi * elk-poc-es
10.0.0.3 28 98 16 0.77 0.87 0.82 mdi - elk-poc-es1