Cluster Health assistance

[ec2 (Gen)]

Hi,

I was hoping to get some assistance with the health of a cluster I am running in a prod environment.

I am relatively new to the Elastic Stack but have done quite a lot of reading on the various aspects of the Elastic Stack configuration (trying to use best practice etc) but there are a lot of elements to consider and it is a bit overwhelming so so any assistance or guidance from an expert would be much appreciated.

Here's our current setup:

• 1 ElasticSearch Node running on a VM in Azure with 4 cores and 32gb memory (8gb heap size). No replica nodes
• 2 Logstash servers running on VMs in Azure. Server 1: 2cores 8gb memory (2gb heap size). Server 2: 2cores 4gb memory (2gb heap size)

Server 1 is running 5 pipelines with the biggest pipeline ingesting a maximum of just under 2k events per second and the others ingesting negligible amounts of data.

We currently have a total of 89 indices and 233 shards. There are daily indices being created for the large pipeline (roughly 50m-100m records in each index. Largest index size is 120gb but average around 50gb). These indices have 5 primary shards each. The other indices have 1 shard each and get rotated monthly (these indices are no bigger than 2gb each).

The health of the cluster is reported to be green in Kibana however, the Elasticsearch node is using over 94% JVM Heap on average and there are frequent garbage collection warnings in Elasticsearch logs such as these:

[2019-03-01T16:10:35,926][WARN ][o.e.m.j.JvmGcMonitorService] [9uxYeLa] [gc][21210] overhead, spent [10s] collecting in the last [10.3s]

Another concern I have is that after restarting Elasticsearch, it takes around 30 minutes to fully recover all shards with lots of errors as such:

[2019-03-01T10:37:00,890][DEBUG][o.e.a.s.TransportSearchAction] [9uxYeLa] All shards failed for phase: [query]

Eventually they all turn green though. Moreover, performing certain queries in Kibana causes an error message saying "x amount of shards failed" even though they all report to be green.

Here's the output of GET _cluster/health/?level=shards?

  {
    "cluster_name" : "elasticsearch",
    "status" : "green",
    "timed_out" : false,
    "number_of_nodes" : 1,
    "number_of_data_nodes" : 1,
    "active_primary_shards" : 233,
    "active_shards" : 233,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 0,
    "delayed_unassigned_shards" : 0,
    "number_of_pending_tasks" : 0,
    "number_of_in_flight_fetch" : 0,
    "task_max_waiting_in_queue_millis" : 0,
    "active_shards_percent_as_number" : 100.0
  }

Here are metric details for the Elasticsearch node for the past 4 hours:

Any indication of what else I can look at? Please let me know if any other information would be helpful.

Gen.

[Ant]

If you only have 1 node then when you cycle it the whole cluster has gone offline, as such if needs to read through all the indexes to activate them. If you ran that same health command when you reboot you would see the active shard percent go from 0 to 100 as active shards goes up and unassigned shards goes down. By default this occurs 2 shards at a time.

Till they have all been loaded then you're cluster won't be avaliable, if that is an issue I'd recommend you add 2 extra nodes to your cluster and mark the zen discovery to 2 in the elasticsearch.yml file. You could add just 1 node but then you risk what's called split brain which if you search for it has been explained many times over by people far better than I could now put it.

In terms of your shards, you don't really want them bigger than 50Gb but if they are smaller than that then there is little to be gained from having many on a single node. There is a lot you can read about that but if you have indexes of 120Gb my feeling woudl be to make those 3 shards rathter than 5. The more shards you have the more JVM heap overhead you have.

Speaking of JVM heap, is there a reason you only have it set to 8GB? Assuming it's not being used for something other than elastic I'd have made it 16GB (which would solve your JVM heap pressure issues) with the other 16GB there for the OS and caching. When increasing JVM heap size do remember that you get good returns up till 32GB after that the cost of managing a JVM that large means you stop seeing benfits to the same degree (I've never tried to see what impact making it higher has)

If you do add extra nodes know that because you have 1 shard nodes you may not see all the perf gains you might as the shards aren't granular enough to be able to distribute the indexes. You could re-index to correct this though.

If I were you though I'd think about rolling out 2 extra nodes but make them half the size of the existing node with 8Gb to the heap (of the 16GB those boxes would have), set replication to 1 on all indexes and then half the resource of the existing box. What may work better rather than halfing the existing VM is to roll out a 3rd VM at half the spec and then decomission the origional one. Doing all that way you get better performance, you can bounce a single node with no loss of service and it only increases the VM cost by 50% to get it. Read up on split brain though so that doesn't bite you in the arse down the line.

[ec2 (Gen)]

@Ant Thanks for your advice!

Makes a lot of sense.

I've increased the JVM heap to 16gb for now and will look at rolling out 2 extra nodes in the near future.

[Ant]

your welcome & good luck!