Elastic Search Crashing and corrupting the index data

Elastic Stack Elasticsearch
1

We are running a 1node Elasticsearch Cluster on Elastic stack version 7.17.9. The node are running on computer with SSD storage and 128GB RAM. But when I indexing some data on elastic-search getting stopped and as well as not able to see old ingested data as well as when I see index status stack management it seen as red, and that index is getting corrupted.
Please help me urgently!
I am only getting the logs as in elastic-search.log file is

[2025-05-27T22:29:00,099][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [PCVITA141] attempting to trigger G1GC due to high heap usage [16373299056]
[2025-05-27T22:29:00,142][INFO ][o.e.i.b.HierarchyCircuitBreakerService] [PCVITA141] GC did bring memory usage down, before [16373299056], after [12694894448], allocations [94], duration [43]

Please help me urgently!

2

What is the output of:

GET /
GET /_cat/nodes?v
GET /_cat/health?v
GET /_cat/indices?v
3 
GET /
Output=>
{
  "name" : "PCVITA141",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "VT1bELb9SKWydPHb2B7rMA",
  "version" : {
    "number" : "7.17.28",
    "build_flavor" : "default",
    "build_type" : "zip",
    "build_hash" : "139cb5a961d8de68b8e02c45cc47f5289a3623af",
    "build_date" : "2025-02-20T09:05:31.349013687Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.3",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
==============================================================
GET /_cat/nodes?v
Output=>
ip        heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
127.0.0.1           78          86  61                          cdfhilmrstw *      PCVITA141
========================================================================
GET /_cat/health?v
Output=>
epoch      timestamp cluster       status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1748368177 17:49:37  elasticsearch red             1         1     14  14    0    1        0             0                  -                 93.3%

=======================================================================
GET /_cat/indices?v
Output=>
health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
red    open   mails                           Ur9L_GN4TVi75rnQsldD4w   1   0                                                  
green  open   folders                         4iJduwc_RKCIKn6fFPTP0g   1   0          7            0     13.8kb         13.8kb
green  open   calls                           EZCWit6TRpes_sCszXLckA   1   0          0            0       227b           227b
green  open   .kibana_7.17.9_001              OsPJPEbhRrmwO8Io8hGLJQ   1   0         22            4      2.3mb          2.3mb
green  open   .apm-agent-configuration        k2iSBYx5Tayc3CA0oNZTNg   1   0          0            0       227b           227b
green  open   .kibana_task_manager_7.17.9_001 MDW2MntlQla8lhCJD9R4gw   1   0         17          243      149kb          149kb
green  open   entity-metadata                 zDnYQwZyS7O7nM0tQ-ip4A   1   0          0            0       227b           227b
4

I'm a details guy, and ...

and

do not match. Looks like you upgraded, maybe by accident?

Your previous post on the forum also did not get a clear resolution, and I'm reminded of the thread title "Using the UpdateByQuery with InlineScript, 100000+ items it is giving the max timeout exception". Er, did you solve that one, if so how, and did you understand what was wrong, and is this the same system? The logs show the garbage collection kicking in.

I understand your frustration, but this is a community forum, the volunteers answering include people like me who are busy looking for work, as well as many people who actually already have work! So, there is no SLA, sorry.

1 Like
5

Yes, in the forum, there is same context question like index corruption, i have founded, they told to check with elastic upgrade 7.17 latest version,
that's why i upgraded.
But still my index is corrupting, my issue is not resolving.

and for my previous error, i changed the way to solve that error.

6

I think we have a much better chance to make progress if you supply a lot more info - including what "changed the way" means specifically. Also I don't quite buy that elasticsearch has got into a state where the dominating (large) index is corrupted, and the only logs are the 2 you shared, neither of which were actually errors or even warnings (note the "INFO").

In that previous thread you wrote "average size of documents: 87100kb", with nesting. I asked on that thread if this was a typo, as the numbers did not add up. You didn't answer.

This was the recommendation based on info you supplied then:

Note that updating very large documents with large number of nested documents results in a lot of overhead as the full document needs to be reindexed. Each nested document is behind the scenes stored as a separate document, which also adds to the overhead. If the numbers are correct and that is your average document size I would recommend you revise how you index and manage your data.

Did you follow this? Is this even the same "data" as that other thread ?

Also in that other thread, you mentioned the "c drive". One way that elasticsearch indices can get corrupted is if something else is "messing" with its data directory. Like anti-virus/malware software. And "user error". That can happen in different OSes,. but seems more common in Windows in my experience.

But you are IMO not supplying enough information to help us help you.

[ Well, there's brighter-than-me people who read here too, so maybe they will figure it out based on what you wrote. ]

1 Like
7

means now I updating the batch batch items.
But for now this is not the case. cause, i am not updating items now.
But, Now i am adding/inserting documents in the Elasticsearch specific index, not updating.
and Elastic is running in the C drive.
and machine has microsoft windows defender is there

and a one document ingesting size is = 134 KB

8

If you are seeing issues with index corruption I suspect this might be the problem. I am not a Windows user, but you need to make sure antivirus does not impact the Elasticsearch data folder. You should be able to see more details around the corruption in the logs though.

In order to verify that this is the issue, you could spin up a cloud instance running Linux and try the same things there and see if the issues go away.

9

@rutuja Specifically, please check Windows / Windows Defender logs for anything which references any files in elasticsearch's data directory.