Elasticsearch 7.17.28 Critical Bugs

Elastic Stack Elasticsearch
1

Anyone still using Elasticsearch 7.17.28
I'm facing some issues and would like to see if it's a elasticsearch issue or something else

Issues like

  1. Snapshot repo permission issue

Desc:
Elasticsearch loses repo access and gains it automatically after some time causing missed snapshots

  1. Client side connection dropping needing restart which further causes downtimes

if anyone faced similar or any elasticsearch side issue in this version (7.17.28)

Do share please

2

Hi Abdullah_Shah, welcome to the Elastic community!

Based on the information shared so far, I would be careful before classifying this as a confirmed Elasticsearch 7.17.28 bug. Both symptoms can happen because of Elasticsearch, but they can also be caused by repository backend, permissions, storage, network, client configuration, or cluster instability.

For the snapshot repository issue, I would first validate the repository itself and the path from every relevant Elasticsearch node to that repository. Elasticsearch verifies that a repository is available and functional on the master and data nodes, so an intermittent failure usually needs to be checked from the repository/backend side as well, not only from the Elasticsearch version side.

Official reference:

A few important things to confirm:

  1. What repository type are you using? fs, s3, azure, gcs, hdfs, etc.
  2. Is this self-managed, ECK, ECE, or Elastic Cloud?
  3. Are multiple clusters using the same repository? If yes, only one cluster should have write access.
  4. Do all master/data nodes have the same access to the repository?
  5. Are there any IAM, service account, temporary credential, mount, NFS, firewall, proxy, or object storage endpoint changes around the failure time?
  6. What exact error appears in the Elasticsearch logs when the snapshot fails?

I would run these checks when the repository is healthy and again when it is failing:

GET _snapshot
GET _snapshot/<repo_name>
POST _snapshot/<repo_name>/_verify
GET _slm/status
GET _slm/policy/*?human=true
GET _slm/stats?human=true

For repeated SLM failures, the Elastic troubleshooting guide recommends checking the affected SLM policy and then checking the elected master node logs during the snapshot execution window.

Official reference:

For the client-side connection issue, I would separate two scenarios:

  1. Application/client HTTP connections to Elasticsearch are dropping.
  2. Elasticsearch nodes are leaving/rejoining the cluster internally.

If this is the application/client side, please share the client language and version, the full exception/stack trace, whether there is a load balancer/proxy/firewall between the app and Elasticsearch, and the client timeout/retry configuration.

If this is Elasticsearch node-to-node instability, the elected master logs are the best starting point. Look for messages such as NodeLeftExecutor and NodeJoinExecutor, and check whether the reason is disconnected, lagging, followers check retry count exceeded, or joining after restart.

Official reference:

I would also check cluster pressure around the same time, especially JVM memory pressure, GC overhead, CPU, disk latency, rejected thread pools, and node restarts. A client may experience connection drops or timeouts when the cluster or one of the target nodes is overloaded.

Useful checks:

GET _cluster/health?pretty
GET _cat/nodes?v&h=name,ip,roles,master,heap.percent,ram.percent,cpu,load_1m
GET _nodes/stats?filter_path=nodes..name,nodes..jvm.mem.pools.old,nodes..thread_pool..rejected

One additional point: 7.17.x is already past its maintenance and support window according to Elastic’s version policy, so even if the immediate root cause is repository/network/client related, I would still include an upgrade plan in the remediation path.

Official reference:

So, based on the current description, my first hypothesis would be:

  • Snapshot issue: intermittent repository/backend/permission/network/access problem.
  • Client connection issue: client/network/load balancer/proxy configuration or cluster pressure/instability.

But to confirm that, the exact repository type, deployment type, Elasticsearch logs, SLM policy output, client error stack trace, and cluster health around the failure time would be needed.

3

Thank you Rafa_SIlva for your detailed help.

and Yes repo is nfs and write-only to one cluster

Also, it is self managed

i further investigated the issue and it was found to be a node restart, an OS issue which was later solved by the team

But recently i faced another similar issue where snapshot failed as partial because one node shard failed due to permission issue of repo

In both scenarios one common factor was change of master
that happened few min-hours before these problems

My question is

Is it necessary to refresh permissions of elasticsearch on backup repo when nodes rejoin the cluster
because this file permission error solves on its own when we just open the vm to check the permissions

never had to regrant the permissions so if they never went away why elasticsearch refuses and throws the error

4

Hi Abdullah_Shah,

No, you should not need to refresh Elasticsearch permissions when a node rejoins the cluster.

For an fs repository over NFS, Elasticsearch depends on the filesystem access available to the OS user running Elasticsearch. If the issue disappears when someone opens/checks the VM, I would first suspect the OS/NFS layer: mount state, stale NFS handle, UID/GID mapping, SELinux, or Elasticsearch starting before the NFS mount is ready.

The repository path must be mounted in the same location on all master/data nodes and allowed in path.repo:

The master change is a useful clue. After a master change, a different master-eligible node may coordinate repository operations. If that node has a different NFS or permission state, snapshot verification can fail.

I would compare the affected node and the new master with simple OS checks, then run:

POST _snapshot/<repo_name>/_verify

If _verify fails only sometimes, I would treat this as intermittent repository accessibility from one or more nodes, not as a permission refresh needed inside Elasticsearch.

5

Yeah i do also think it's a stale NFS issue.

While i investigate it's root cause further
I think the best solution right is to either make nfs pre boot check for Elasticsearch or just refresh permissions when doing maintenance on ecs nodes.

Well anyways,

Thank you for your time and valuable knowledge.

Stay blessed.