Elasticsearch 7.16.2 Snapshot Status Error: 'duration cannot be negative'

[Anastasia_Kornienko (Аnastasia Kornienko)]

Hello,

I'm encountering an error when trying to check the status of a snapshot. The situation is as follows:

I have an Elasticsearch 7.16.2 cluster consisting of 5 nodes. One node is currently unavailable while a snapshot is being taken. When I attempt to check the snapshot status using the following command:

curl -X GET "localhost:9200/_snapshot/_status?pretty"

I get the following error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "duration cannot be negative, was given [-5349206]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason": "duration cannot be negative, was given [-5349206]"
  },
  "status" : 400
}

After some time, when the backup process completes, the snapshot status command works correctly.

Why does this error occur?

[piergm (Matteo Piergiovanni)]

Hello @Anastasia_Kornienko,
from the docs you can add the optional query parameter ignore_unavailable=true to your request and get more detailed information on what the failure is under the failures section of the response.
That should provide a human readable failure that I can help you resolve instead of an error.
Hope this helps!

[DavidTurner]

You could also pass the ?error_trace query parameter to get a stack trace which would show exactly where this exception is happening. However one possible explanation for this would be if you have nodes in your cluster with wildly incorrect local clocks. Try GET _nodes/_all/stats/,?filter_path=nodes.*.timestamp,nodes.*.name to see the current time in milliseconds on each node and look for outliers there.