Hung tasks in ES, that cannot be cancelled using the Task API

Elastic Stack Elasticsearch
1

I have an on-going problem with accumulating read/search tasks that appear to be stuck.
The included picture shows the situation, where the 02 index has accumulated ~70 search tasks that never go away.

image
image1909×299 84.5 KB

The tasks are of type netty or transform, and the action is reads/search.

{
    "aDhN0FiNQnW4q7UiY8qAfw:8363529459" : {
        "node" : "aDhN0FiNQnW4q7UiY8qAfw",
        "id" : 8363529459,
        "type" : "netty",
        "action" : "indices:data/read/search[phase/query]",
        "description" : "shardId[[02][33]]",
        "start_time_in_millis" : 1579507610392,
        "running_time_in_nanos" : 45661052782486,
        "cancellable" : true,
        "parent_task_id" : "fvp4CpvsSl6cvb6IeTxlag:4187545907"
    }
}

{
    "C2sZvd9MQM-trg7ShfzAig:3580000762" : {
        "node" : "C2sZvd9MQM-trg7ShfzAig",
        "id" : 3580000762,
        "type" : "transport",
        "action" : "indices:data/read/search",
        "description" : "indices[02], types[], search_type[QUERY_THEN_FETCH], source[...]",
        "start_time_in_millis" : 1579248693084,
        "running_time_in_nanos" : 178537370554429,
        "cancellable" : true
    }
}

Using the Task Management API to "cancel" these tasks, does nothing.
The master nodes and coordinating nodes have been restarted, and that didn't do anything either.
This is happening on Elasticsearch 5.6.x

It appears that the source of the problem is that the query took too long to respond, and the client timed out.

Does anyone have ideas on how this could be resolved?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.