My Elasticsearch experiences freezing 3 to 4 times a day

Elastic Stack Elasticsearch
1

Hello everyone,

I'm currently facing an issue that's not well defined.

111
1111290×387 170 KB

. The freezing periods last approximately 30 to 60 seconds. During these periods, I'm unable to query it using Kibana or the Nodejs client. I've monitored the system, and I can see interruptions as shown in the graph.

There's nothing unusual in the logs. Has anyone in the community encountered this problem before and could provide guidance on how to resolve it?

2

What are the configuration of your cluster? How many nodes do you have?

You need to provide more context.

3

So sorry. Here my info:
Custer health:

{
  "cluster_name": "elasticsearch",
  "status": "yellow",
  "timed_out": false,
  "number_of_nodes": 1,
  "number_of_data_nodes": 1,
  "active_primary_shards": 52,
  "active_shards": 52,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 30,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 63.41463414634146
}

Node and Server Information

01
012821×1297 152 KB

4

So you have a single-node cluster running on a machine with ~ 32 GB/10 vCPU.

What is the disk type? SSD or HDD? Your CPU utilization is low, but your load is pretty high in some moments, this suggests thar you have some I/O issues.

Did you configure Elasticsearch JVM Heap or are you using the default (which is 50%)?

Also, are you running anything else in this same server? And what is the version you are using?

5

I don't sure Hard disk type. With lsblk command is show HDD but on seller page they said SSD (netcup)

lsblk -d -o name,rota,size,type
NAME ROTA  SIZE TYPE
sda     1    2T disk
sr0     1 1024M rom

Did you configure Elasticsearch JVM Heap or are you using the default (which is 50%)?
No, i use default config.

Also, are you running anything else in this same server?
I installed mariadb but it's not used much

And what is the version you are using?
And here is information:

{
  "name" : "v2202212170493212509",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "00001",
  "version" : {
    "number" : "8.5.3",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "4ed5ee9afac63de92ec98f404ccbed7d3ba9584e",
    "build_date" : "2022-12-05T18:22:22.226119656Z",
    "build_snapshot" : false,
    "lucene_version" : "9.4.2",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
6

Here about JVM Heap

112233
1122333609×1565 419 KB

I see that the problem still occurs when the CPU is below 20%.
Maybe as you said, the problem lies with the hard disk

7

Yeah, low cpu but high load is an indication of high iowait, in this image you shared the CPU image stays pretty low, below 20%, but the load reached almost 60 at one point, which is an issue since you have only 10 vCPU.

HDD drives can impact a lot in Elasticsearch performance, you need to check this.

What is the filesystem, xfs or ext4/3? If it is ext4 maybe you can try to improve things a little changing some mount parameters.

8

What is the filesystem, xfs or ext4/3?
/dev/sda3 ext4 2.0T 533G 1.4T 28% /
/dev/sda2 ext4 974M 198M 709M 22% /boot
I saw ext4, how can i improve it ?

9

You need to check if the mount options already include the noatime option, if not, you can add it, this can give you a little boost in performance, but it is not guaranteed that it will solve your issue since HDD disks are still not good for Elasticsearch.

But please read about it first because any mistake can impact your system and put it on a state where the system may not be able to boot.

One reference article is this one.

10

Thank you very much, your advice is very good

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.