Delaying Tasks

[Emiliano_Baum]

Hi, i try to set the delayed_timeout variable and have an 503 master timout error. Before that the cluster start to delay tasks:
"number_of_pending_tasks" : 127,

the command I ran was:

curl -X PUT "localhost:9200/_all/_settings?pretty" -H 'Content-Type: application/json' -d' {"settings": {"index.unassigned.node_left.delayed_timeout": "5m" }}'

When i check the pending task i found:

"tasks" : [
{
  "insert_order" : 165407269,
  "priority" : "URGENT",
  "source" : "update-settings",
  "executing" : true,
  "time_in_queue_millis" : 522085,
  "time_in_queue" : "8.7m"
},

is rare, because its suppose to be rejected...

On the logs of MASTER:

    [2021-01-22T17:35:10,899][DEBUG][o.e.a.a.i.s.p.TransportUpdateSettingsAction] [CGSS-CLUSTER01-MASTER-2] failed to update settings on indices ...

strange...

[Christian_Dahlqvist]

It would help if you told us a bit more about the cluster and the use case. What is the full output of the cluster stats API?

[Emiliano_Baum]

Hi Christian, I finally unplugged the master node in charge the moment I ran the command and all pending tasks are gone.
At the time of this issue, a rolling update from 6.6.0 to 6.8.13 was run.