So lets say I have 100 data nodes where I installed https://github.com/elastic/elasticsearch-hadoop/tree/master/repository-hdfs to send my snapshots to hdfs.
My HDFS install is secured by kerberos. Before I do a snapshot, am I going to have to kinit to get a kerberos ticket on every single one of those data nodes?
As far as I know (I haven't used Kerberos in a while) yes. The reason behind it is that the Hadoop configuration and thus the HDFS client connection doesn't hold any Kerberos information - this is installed at system level.
There are various extensions or tooling out there to automate this but at the end of the day is something that sits outside Hadoop/HDFS and thus ES hadoop as well.
Of course, you can try to do it without the authentication but likely, without the relevant ticket, any HDFS activity will fail.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.