Elasticsearch snapshot repository fails validation

Elastic Stack Elasticsearch
25

For anyone following along, the problem actually has to do with SystemD implementing a per-process /tmp space that take precedence over anything else that might be mounted in that space. As you can see from grepping my elasticsearch's process mounts, it is given it's own /tmp dir:

root@facility-es01:~# grep /tmp /proc/11500/mounts
/dev/vda1 /tmp ext4 rw,relatime,errors=remount-ro 0 0
/dev/vda1 /var/tmp ext4 rw,relatime,errors=remount-ro 0 0
backup0.ftc:/pool0/elasticsnapshots /tmp/elastic nfs rw,sync,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,noac,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.40.8.12,mountvers=3,mountport=36300,mountproto=udp,lookupcache=none,local_lock=none,addr=10.40.8.12 0 0

This meant that any writes to /tmp/elastic/ were not going to my NFS mount, as that mount was being clobbered by the per-process /tmp feature of SystemD. Moving my NFS mount into /mnt/ (where it probably should be anyway) fixes this issue and snapshot repo verification works properly:

root@facility-kibana:~# curl -s -XPOST 'localhost:9200/_snapshot/backup0/_verify'| jq
{
  "nodes": {
    "_AAzUpbdSLmH8Eu2fATC8w": {
      "name": "facility-es03.ftc"
    },
    "PFSVaur8STWZap533rLEXg": {
      "name": "facility-es01.ftc"
    },
    "gUK92CyURGKaIh1AYYxFUA": {
      "name": "facility-es02.ftc"
    }
  }
}

I want to thank David Turner for his time looking into this issue and I appreciate his expertise.