I'm looking for the Elastic best practises/supported configurations for shared filesystem with specific regard to performing snapshot backups.
NFS shared filesystem works and most people I believe uses this.
but you can use Azure, Google Cloud, HDFS as well
Am really looking for a documented list of supported configurations rather than using the suck it and see approach.
There isn't a list of supported configurations since there's just so many possibilities, and shared filesystems are all similar enough that it doesn't really matter to Elasticsearch. NFS is a popular choice, so use that if you can. Maybe we can turn this around: what configuration are you planning to use? Is it NFS? If not, why not?
NFS would actually be my last choice - http or even a shared VMDK (iSCSI) would be my preferences.
Minio is a popular (supported and tested) choice for a HTTP(S) endpoint that is compatible with the repository-s3 plugin for taking snapshots. Other S3-compatible endpoints are also normally ok, but we only test against Minio and S3.
I don't think writing directly to a shared VMDK is officially supported (i.e. it's not tested) and I have no experience with them. I had a brief look for docs on their consistency semantics but couldn't find any, and this is normally an important deciding factor in whether things work correctly or not. As long as it's more strongly consistent than S3 (not a high bar) then I'd guess it'd be ok, but I don't think that's enough for you.
So . - where is the defacto list of what IS supported (that's really what i'm looking for). Elastic seem to be saying that some things are supported and some are potentially not and yet not advertising (at least nowhere I can find) - what that list consists of - which is a bit of a shoulder slope to say the least.
1 Like
As I said, there is no such list. In practice I don't think I've ever heard of a production-ready shared filesystem that didn't work for snapshots, but you seem to be asking for stronger assurances than this.
I also think we might be using the word "supported" to mean different things. No matter how weird your configuration we'll do our best to support you through any issues, up to and including helping you to identify bugs in third-party components. In contrast, it is impossible to call something "officially supported" without evidence, through our own testing, that it actually works correctly. There are far too many shared filesystem configurations for it to be practical to test any significant fraction of them, and we see so few issues with the different implementations that it's hard to justify expanding the testing in this area.
You must bear in mind that this is a free community forum where we can offer a certain amount of advice and guidance, but ultimately it's your system, your data, and your responsibility. You can stick with popular tried-and-tested configs, or you can go for something else, it's up to you. If you want some stronger assurances, perhaps the official support and/or consulting services are what you are looking for?
OK so I'd like to use either a simple HTTP web URL (Apache)
or
Red Hat's Ceph S3
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.