Size difference for snapshot repository when executing Disk Free (DF) vs Disk Usage (DU) commands on Linux

[JayNair]

We are facing an issue with the disk usage on the NFS mounted on the Elasticsearch VMs for storing snapshots.

We create daily indices and snapshots are created for indices older than 30 days and retain the snapshots for 30 more days.

Curator is used for creating snapshots, deleting indices as well as deleting snapshots older than 30 days. We have those are scheduled jobs.

NFS storage is 4 TB

The issue we have now is, when I check free disk space using “df -kh”, it says 97% or 3.8 TB disk is used.

But when I do a Disk Usage using “du –h” on the mount, I get 400 GB. Same if I try to get the size through WinSCP.

Below threads explains how Elasticsearch creates snapshots and some segments could still stay on the disk even if you delete the snapshot if those are referenced by other snapshots.

Curator delete snapshots

Sorry not to explain me clearly. I mean: I create a filesystem with a folder with a month name, so, every two months I delete the snapshots via API, but the files remain in the disk, so I delete the folder physically. If I delete the snapshots with curator, it deletes also the files in the disk?

Delete Snaphost can't reduce disk usage

It basically means that snapshots you deleted were referring segments that are still used by other snapshots.

Does the live segments of deleted snapshots stay hidden and won’t count when doing Disk Usage? Is that the reason for the difference in size when doing df vs du?

Also can the size of those segments realistically be 8 times higher than the current snapshots to justify 400 GB vs 3.8 TB?