is there any option to backup elasticsearch direct to tape as shared file system will require equal to all index size ?
No. The only safe and supported way to back up a running Elasticsearch index is via the Snapshot and Restore API.
Snapshotting brings all shards of an index into the same location, the repository, after the shards (Lucene) have been informed of the snapshot being in progress. If you simply try to copy the raw lucene segments, you'll likely end up with corrupted shards, as Lucene may be doing background tasks. In addition, as Elasticsearch is responsible for the assignment of shards to nodes, there is no guarantee that a shard belongs to the same node later on.
thanks for your update. i dont have space to backup. so i will create repository for each index and backup each index and delete it after backup
if i backup an index . will each node write its shard into shared mount point
Yes, that's correct. During a full cluster snapshot, all shards from all nodes are written to the shared repository. If backing up individual indices, all shards from these indices are written there.
thanks for update. one final question. does the shared file system should be NFS .
and suppose if we have 100 nodes , then this shared file system mount across all 100 nodes.
will it not exceed storage or os limitation of maximum share ? what kind of file system zfs/ext3 ?
It has to be a shared filesystem if you use filesystem based repositories. Typically this would be implemented with NFS, but other network filesystems might work, too. The same filesystem must be available on all nodes of the cluster.
There are alternatives to filesystem based repositories, such as S3, GCS, etc. With these, the chosen service provides the disk space, which of course has to be provisioned big enough for the data held in a snapshot.
Hope this helps. Good luck snapshotting!
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.