Array index out of bound exception

Hello! i have snapshot repository which is located on NFS share, and i'm try to register repo from my local ES instance and got strange error

curl  -X PUT "192.168.2.11:9200/_snapshot/delete_after_read?pretty" -H 'Content-Type: application/json' -d'
 {
   "type": "url",
   "settings": {
     "url": "file:/opt/elasticsearch-backup/elastique-backup"
   }
 }
 '

{
  "error" : {
    "root_cause" : [
      {
        "type" : "array_index_out_of_bounds_exception",
        "reason" : "Index 0 out of bounds for length 0"
      }
    ],
    "type" : "array_index_out_of_bounds_exception",
    "reason" : "Index 0 out of bounds for length 0"
  },
  "status" : 500
}

why this error occured?

Can you share the output of that call when adding the parameter error_trace=true and share the whole output in a gist?

That said, how about using the fs repository instead? Alternatively you may want to try file:///opt/... for the url.

1 Like

Shouldn't you be using;

curl -X PUT "host:9200/_snapshot/delete_after_read?pretty" -H 'Content-Type: application/json' -d'
{
  "type": "fs",
  "settings": {
    "location": "/opt/elasticsearch-backup/elastique-backup",
  }
}

As per Register a snapshot repository | Elasticsearch Guide [7.14] | Elastic?

With error trace enabled

 curl  -X PUT "192.168.2.11:9200/_snapshot/delete_after_read?pretty&error_trace=true" -H 'Content-Type: application/json' -d'
 {
   "type": "url",
   "settings": {
     "url": "file:/opt/elasticsearch-backup/elastique-backup"
   }
 }
 '

{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[delete_after_read] path  is not accessible on master node",
        "stack_trace" : "RepositoryVerificationException[[delete_after_read] path  is not accessible on master node]; nested: ArrayIndexOutOfBoundsException[Index 0 out of bounds for length 0];\nCaused by: java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0\n\tat org.elasticsearch.common.Numbers.bytesToLong(Numbers.java:37)\n\tat org.elasticsearch.common.Numbers.bytesToLong(Numbers.java:44)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.readSnapshotIndexLatestBlob(BlobStoreRepository.java:2556)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2547)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1648)\n\tat org.elasticsearch.repositories.RepositoriesService$4.doRun(RepositoriesService.java:392)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:831)\n"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[delete_after_read] path  is not accessible on master node",
    "caused_by" : {
      "type" : "array_index_out_of_bounds_exception",
      "reason" : "Index 0 out of bounds for length 0",
      "stack_trace" : "java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0\n\tat org.elasticsearch.common.Numbers.bytesToLong(Numbers.java:37)\n\tat org.elasticsearch.common.Numbers.bytesToLong(Numbers.java:44)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.readSnapshotIndexLatestBlob(BlobStoreRepository.java:2556)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2547)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1648)\n\tat org.elasticsearch.repositories.RepositoriesService$4.doRun(RepositoriesService.java:392)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:831)\n"
    },
    "stack_trace" : "RepositoryVerificationException[[delete_after_read] path  is not accessible on master node]; nested: ArrayIndexOutOfBoundsException[Index 0 out of bounds for length 0];\nCaused by: java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0\n\tat org.elasticsearch.common.Numbers.bytesToLong(Numbers.java:37)\n\tat org.elasticsearch.common.Numbers.bytesToLong(Numbers.java:44)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.readSnapshotIndexLatestBlob(BlobStoreRepository.java:2556)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2547)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1648)\n\tat org.elasticsearch.repositories.RepositoriesService$4.doRun(RepositoriesService.java:392)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:831)\n"
  },
  "status" : 500
}
{
  "type": "fs",
  "settings": {
    "location": "/opt/elasticsearch-backup/elastique-backup",
    "compress": true
  }
}
'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "parsing_exception",
        "reason" : "Failed to parse object: expecting token of type [null] but found [VALUE_NUMBER]",
        "line" : 1,
        "col" : 9085,
        "stack_trace" : "ParsingException[Failed to parse object: expecting token of type [null] but found [VALUE_NUMBER]]\n\tat org.elasticsearch.common.xcontent.XContentParserUtils.parsingException(XContentParserUtils.java:74)\n\tat org.elasticsearch.common.xcontent.XContentParserUtils.ensureExpectedToken(XContentParserUtils.java:68)\n\tat org.elasticsearch.repositories.RepositoryData.snapshotsFromXContent(RepositoryData.java:819)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.getRepositoryData(BlobStoreRepository.java:2099)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1906)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:831)\n"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[delete_after_read] Unexpected exception when loading repository data",
    "caused_by" : {
      "type" : "parsing_exception",
      "reason" : "Failed to parse object: expecting token of type [null] but found [VALUE_NUMBER]",
      "line" : 1,
      "col" : 9085,
      "stack_trace" : "ParsingException[Failed to parse object: expecting token of type [null] but found [VALUE_NUMBER]]\n\tat org.elasticsearch.common.xcontent.XContentParserUtils.parsingException(XContentParserUtils.java:74)\n\tat org.elasticsearch.common.xcontent.XContentParserUtils.ensureExpectedToken(XContentParserUtils.java:68)\n\tat org.elasticsearch.repositories.RepositoryData.snapshotsFromXContent(RepositoryData.java:819)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.getRepositoryData(BlobStoreRepository.java:2099)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1906)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:831)\n"
    },
    "stack_trace" : "RepositoryException[[delete_after_read] Unexpected exception when loading repository data]; nested: ParsingException[Failed to parse object: expecting token of type [null] but found [VALUE_NUMBER]];\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1960)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:831)\nCaused by: ParsingException[Failed to parse object: expecting token of type [null] but found [VALUE_NUMBER]]\n\tat org.elasticsearch.common.xcontent.XContentParserUtils.parsingException(XContentParserUtils.java:74)\n\tat org.elasticsearch.common.xcontent.XContentParserUtils.ensureExpectedToken(XContentParserUtils.java:68)\n\tat org.elasticsearch.repositories.RepositoryData.snapshotsFromXContent(RepositoryData.java:819)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.getRepositoryData(BlobStoreRepository.java:2099)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1906)\n\t... 6 more\n"
  },
  "status" : 500
}

Is this repository supposed to be readonly or not? See Register a snapshot repository | Elasticsearch Guide [7.14] | Elastic

Yeah! it supposed to be readonly, but i have strange errors with registering

And almost same thing when i register another repository ( which located on the same NFS share )

curl  -X PUT "192.168.2.11:9200/_snapshot/forteid?pretty&error_trace=true" -H 'Content-Type: application/json' -d'
> {
>   "type": "url",
>   "settings": {
>     "url": "file:/opt/elasticsearch-backup/prod-forteid-efk-7.4.1"
>   }
> }
> '
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_exception",
        "reason" : "[forteid] Could not determine repository generation from root blobs",
        "stack_trace" : "RepositoryException[[forteid] Could not determine repository generation from root blobs]; nested: ArrayIndexOutOfBoundsException[Index 0 out of bounds for length 0];\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1884)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:831)\nCaused by: java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0\n\tat org.elasticsearch.common.Numbers.bytesToLong(Numbers.java:37)\n\tat org.elasticsearch.common.Numbers.bytesToLong(Numbers.java:44)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.readSnapshotIndexLatestBlob(BlobStoreRepository.java:2556)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2547)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1881)\n\t... 6 more\n"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[forteid] Could not determine repository generation from root blobs",
    "caused_by" : {
      "type" : "array_index_out_of_bounds_exception",
      "reason" : "Index 0 out of bounds for length 0",
      "stack_trace" : "java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0\n\tat org.elasticsearch.common.Numbers.bytesToLong(Numbers.java:37)\n\tat org.elasticsearch.common.Numbers.bytesToLong(Numbers.java:44)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.readSnapshotIndexLatestBlob(BlobStoreRepository.java:2556)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2547)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1881)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:831)\n"
    },
    "stack_trace" : "RepositoryException[[forteid] Could not determine repository generation from root blobs]; nested: ArrayIndexOutOfBoundsException[Index 0 out of bounds for length 0];\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1884)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:831)\nCaused by: java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0\n\tat org.elasticsearch.common.Numbers.bytesToLong(Numbers.java:37)\n\tat org.elasticsearch.common.Numbers.bytesToLong(Numbers.java:44)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.readSnapshotIndexLatestBlob(BlobStoreRepository.java:2556)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2547)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1881)\n\t... 6 more\n"
  },
  "status" : 500
}

Guys, any ideas to fix that?

can you try the file:/// suggestio I mentioned above and tell if it worked?

curl  -X PUT "192.168.2.11:9200/_snapshot/delete_after_read?pretty" -H 'Content-Type: application/json' -d'
 {
   "type": "url",
   "settings": {
     "file:///": "file:/opt/elasticsearch-backup/elastique-backup"
   }
 }
 '

like this?

No, not in the key, but in the value of that field.

@spinscale same error with file:///

curl  -X PUT "192.168.2.11:9200/_snapshot/lento4ka?pretty&error_trace=true" -H 'Content-Type: application/json' -d'
{
  "type": "url",
  "settings": {
    "url": "file:///opt/lenta/efk-anthill-nopci-14.03.2021"
  }
}
'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_exception",
        "reason" : "[lento4ka] Could not determine repository generation from root blobs",
        "stack_trace" : "RepositoryException[[lento4ka] Could not determine repository generation from root blobs]; nested: ArrayIndexOutOfBoundsException[Index 0 out of bounds for length 0];\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1884)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:831)\nCaused by: java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0\n\tat org.elasticsearch.common.Numbers.bytesToLong(Numbers.java:37)\n\tat org.elasticsearch.common.Numbers.bytesToLong(Numbers.java:44)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.readSnapshotIndexLatestBlob(BlobStoreRepository.java:2556)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2547)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1881)\n\t... 6 more\n"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[lento4ka] Could not determine repository generation from root blobs",
    "caused_by" : {
      "type" : "array_index_out_of_bounds_exception",
      "reason" : "Index 0 out of bounds for length 0",
      "stack_trace" : "java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0\n\tat org.elasticsearch.common.Numbers.bytesToLong(Numbers.java:37)\n\tat org.elasticsearch.common.Numbers.bytesToLong(Numbers.java:44)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.readSnapshotIndexLatestBlob(BlobStoreRepository.java:2556)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2547)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1881)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:831)\n"
    },
    "stack_trace" : "RepositoryException[[lento4ka] Could not determine repository generation from root blobs]; nested: ArrayIndexOutOfBoundsException[Index 0 out of bounds for length 0];\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1884)\n\tat org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:831)\nCaused by: java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0\n\tat org.elasticsearch.common.Numbers.bytesToLong(Numbers.java:37)\n\tat org.elasticsearch.common.Numbers.bytesToLong(Numbers.java:44)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.readSnapshotIndexLatestBlob(BlobStoreRepository.java:2556)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2547)\n\tat org.elasticsearch.repositories.blobstore.BlobStoreRepository.doGetRepositoryData(BlobStoreRepository.java:1881)\n\t... 6 more\n"
  },
  "status" : 500
}

hi guys, any ways to solve this?

:innocent: is there any solution?