Architecture Validation: Backing up ES Clusters to Local Disk without NAS (using S3 Gateway sidecar)

Elastic Stack Elasticsearch
1

The Context I am a C++ developer working on an on-premise Visual Analytics application. Our service (DocumentStore) manages an Elasticsearch cluster (handling logs, events, configuration). We need to implement a Backup & Restore feature.

The Constraints (Why this is hard) We deploy into strict on-prem environments (often air-gapped) on both Windows and Linux.

  1. No NAS Guarantee: We cannot force customers to provision a shared network drive (NFS/SMB) for the cluster.

  2. Portability (The "Tarball" Requirement): Users need to click a button, generate a backup, and download a single .tar.gz/.zip file. They might move this file to a different environment to restore it.

  3. Cluster Support: We must support multi-node ES clusters.

The Problem Since we don't have a shared filesystem, we can't use the standard fs repository type. If we point Node A and Node B to a local path (e.g., C:\Backups), the Master Node can't verify the integrity of the snapshot because the files are split across physical machines ("split-brain" filesystem).

The Proposed Solution: "The Local S3 Sidecar" To solve the "No NAS" problem, I am planning to bundle a lightweight S3-compatible server (sidecar) alongside our C++ service.

  1. The Setup: Our service launches an S3 gateway (e.g., SeaweedFS or Rclone) listening on port 8333 on the local machine.

  2. The Config: We configure Elasticsearch (via repository-s3) to write to http://<Our_Service_IP>:8333.

  3. The Result: ES streams data over TCP. It thinks it is writing to the cloud, but it's actually writing to our service's local disk.

  4. The Export: When the user clicks "Download," our service zips the local data directory and serves it.

The Dilemma: Rclone vs. SeaweedFS I am debating between two tools for this "Sidecar" role and would love community feedback:

  • Option A: Rclone (rclone serve s3)

    • Pros: It writes plain, transparent files to disk (1:1 mapping with ES Snapshot files). This makes zipping the directory for the "Tarball" requirement instant and simple.

    • Cons: It is a "dumb pipe." No replication. If the node running Rclone dies, that repository is offline.

  • Option B: SeaweedFS

    • Pros: Native replication. I can run it on multiple nodes and have them sync data active-active, providing High Availability for the backup repo.

    • Cons: It stores data as opaque blobs/volumes. To generate the user's "Tarball," I cannot just zip the disk. I have to "export" the files via API to a temp folder first, effectively doubling the I/O and disk space required during download.

My Questions to the Community:

  1. Is this an anti-pattern? Is running an S3 gateway on local disk specifically to bypass the "Shared Drive" requirement for ES Clusters a known recipe for disaster?

  2. Rclone Reliability: Has anyone used rclone serve s3 as a production target for Elasticsearch snapshots? Is the locking/consistency robust enough for ES?

  3. SeaweedFS Overhead: For those using SeaweedFS, is using it strictly as a "Local Filesystem Abstraction" overkill? Is the overhead of exporting files out of the blob format (to create a zip) painful for datasets in the 10GB-100GB range?

Any insights or war stories from similar air-gapped deployments would be greatly appreciated.

2

All nodes need access to the same S3 bucket the same way they all need access to the same shared filesystem for an fs repository, so I do not understand how this would work.

Have you tested it?

When it comes to using custom S3 implementations it is important that they are fully compatible with S3, which is not always the case. I would therefore recommend thorough testing.

3

I think the OP means they’ll run one of these S3 sidecar things for the whole cluster, not one per node. That sounds reasonable to me, although (as Christian says) it does need to be truly S3-compatible. You may be surprised at how badly some implementations deviate from the behaviour of the real S3. I’ve no experience with either Rclone or SeaweedFS particularly. Historically we’ve recommended MinIO but that project just went into maintenance mode (and started closing S3-compatibility bug reports without action) so that seems worth avoiding now. Garage is another open-source option but they don’t seem particularly keen at fixing known deviations from S3 either.

This is all covered here in the manual by the way.

2 Likes
4

Realised that may be the case when I reread it. I would consider having a single instance with a S3 gateway as a different way to provide shared storage, so would not necessarily say it is side-stepping the requirement. :slight_smile:

If that is the case, would it then be possible to instead have a NFS sidecar (or similar) and thereby avoid potential S3 compaibility issues?

1 Like
5

We want to have multiple sidecars per cluster, but not per ES node - as far as ES is concerned, it’s a single endpoint for the whole cluster. The reason is that we not only have ES clustered, but also our services, and quite often ES and our nodes are on different machines. This is why we might need to switch between the endpoints if one them dies, or use Nginx or similar to specify a single one, then do the redirection there.

I checked Garage, but they don’t ship with Windows binaries, which we need. I can compile it myself, but need more investigation whether it’s been tested on Windows. Apparently it’s much lighter than SeaweedFS, and since I’m not looking for a full object storage, just something that acts like S3 but stores data locally, it could do the trick :slight_smile:

As for NFS sidecar - as mentioned above, we need a mechanism to replicate snapshots between our nodes. ES takes a snapshot, stores it in node A → replicate to nodes B, C… if node A goes down, we still have access to the full history. I don’t want to implement this replication logic myself, apparently Seaweed has it implemented already. Maybe there are some NFS providers that offer this functionality, I need to check.

Thanks for the points about S3 compatibility :slight_smile: at the moment I’m just trying to make sure that “tricking” ES into thinking it communicates with S3, whereas under the hood it’s a local storage, isn’t fundamentally wrong. I am going to test it thoroughly, make sure it’s compatible.

6

Yes, can confirm, this is pretty common. The only difficulty most people face seems to be when they settle on something that claims to be S3-compatible but turns out not to be.

7

Brilliant, thank you very much for your help!