Searchable snapshots with index.store.snapshot.cache.enabled:false. throws error

[cks (Charles)]

Testing searchable snapshots on Flash based fast NFS gave errors with cache:false

1. Using a 20 node Elastic cluster version 7.5.0, took snapshot of a 3.2TB index (40 shards, no replicas) and stored in NFS mount.
2. Built Elasticsearch 8.0 from github master branch. Setup on 20 node cluster. Enable trial lic.
3. Registered the NFS repo with path.repo mount directory. Shows the snapshots correctly.
4. Mounted the index from snapshot with index.store.snapshot.cache.enabled:false

   {  
    "index": "apc-original-idx",   
    "renamed_index": "apc-snapshot-idx",   
    "index_settings": 
    {    
     "index.number_of_replicas": 0,  
     "index.store.snapshot.cache.enabled": "false", 
     "index.store.snapshot.cache.prewarm.enabled": "false" }, 
     "ignored_index_settings": [ "index.refresh_interval" ] 
    }'

5. When I search on this mounted index from snapshot, got errors.

    {
          "error" : {
            "root_cause" : [
              {
                "type" : "e_o_f_exception",
                "reason" : "Reading past end of file [position=73972115, length=6940751] for DirectBlobContainerIndexInput{resourceDesc=randomaccess, fileInfo=[name: __QaYB1LDURCSIkt840sPV8g, numberOfParts: 1, partSize: 8192pb, partBytes: 9223372036854775807, metadata: name [_md.nvd], length [34703814], checksum [1c71uzc], writtenBy [8.3.0]], offset=6940794, length=6940751, position=7341534}"
              },
              {
                "type" : "runtime_exception",
                "reason" : "runtime_exception: Invalid vInt detected (too many bits)"
              },
              {
                "type" : "array_index_out_of_bounds_exception",
                "reason" : "Index -79 out of bounds for length 33"
              },
              {
                "type" : "array_index_out_of_bounds_exception",
                "reason" : "Index -64 out of bounds for length 33"
              },  

When I mount from snapshot with index.store.snapshot.cache.enabled:true, search works, however see a lot data written (300-350GB) to hot tier in path.data folder. After each search additional data is written.

Is it possible to make searchable snapshots work without moving large data from path.repo to path.data?

What does setting "index.store.snapshot.uncached_chunk_size": "4k" do? Did not see any change when I set this size.

[DavidTurner]

First up, this feature is deeply experimental right now so you may well encounter bugs. This sounds like a bug. Could you enable debug logging, reproduce the problem, and then share the whole stack trace and any surrounding log messages?

@cks

Is it possible to make searchable snapshots work without moving large data from path.repo to path.data?

Yes, it's possible, but we don't really plan to do so. The focus of this feature is on searches of blob stores (e.g. S3) which offer cheap and reliable storage but for which we have to use temporary storage to work around their terrible latency, poor random access performance, and request-based pricing. NFS doesn't really have these characteristics so it doesn't need these workarounds. You may as well just use normal indices on your NFS volume and avoid all the overhead.