Why do I see multiple search tasks for a single search in Kibana

As the title suggests when I run a query in Kibana using the discover app I see two identical search actions using the task management API. Just curious why this is and if it is actually running the same query twice against my ES cluster

Example:
Discover > filebeat-hdfs-production-* Index Pattern (All HDFS logs) > Last 30 days (Rounded)

Tasks management API

    _tasks?actions=*search&pretty&detailed"

    {
      "nodes" : {
        "WKr4WGjVQTSKGJTVWHd0OA" : {
          "name" : "<redacted>",
          "transport_address" : "<redacted>:9300",
          "host" : "<redacted>",
          "ip" : "<redacted>:9300",
          "roles" : [ ],
          "attributes" : {
            "xpack.installed" : "true"
          },
          "tasks" : {
            "WKr4WGjVQTSKGJTVWHd0OA:23856284" : {
              "node" : "WKr4WGjVQTSKGJTVWHd0OA",
              "id" : 23856284,
              "type" : "transport",
              "action" : "indices:data/read/search",
              "description" : "indices[filebeat-hdfs-production-ilm-alias,filebeat-hdfs-production-2020.03.19-000002,filebeat-hdfs-production-2020.03.19-000003], types[], search_type[QUERY_THEN_FETCH], source[{\"size\":500,\"timeout\":\"30000ms\",\"query\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"from\":\"2020-04-12T23:00:00.000Z\",\"to\":\"2020-05-13T15:04:26.467Z\",\"include_lower\":true,\"include_upper\":true,\"format\":\"strict_date_optional_time\",\"boost\":1.0}}}],\"filter\":[{\"match_all\":{\"boost\":1.0}}],\"adjust_pure_negative\":true,\"boost\":1.0}},\"version\":true,\"_source\":{\"includes\":[],\"excludes\":[]},\"stored_fields\":\"*\",\"docvalue_fields\":[{\"field\":\"@timestamp\",\"format\":\"date_time\"},{\"field\":\"event.created\",\"format\":\"date_time\"},{\"field\":\"event.end\",\"format\":\"date_time\"},{\"field\":\"event.start\",\"format\":\"date_time\"},{\"field\":\"file.ctime\",\"format\":\"date_time\"},{\"field\":\"file.mtime\",\"format\":\"date_time\"}],\"script_fields\":{},\"sort\":[{\"@timestamp\":{\"order\":\"desc\",\"unmapped_type\":\"boolean\"}}],\"track_total_hits\":2147483647,\"aggregations\":{\"2\":{\"date_histogram\":{\"field\":\"@timestamp\",\"time_zone\":\"Europe/London\",\"fixed_interval\":\"12h\",\"offset\":0,\"order\":{\"_key\":\"asc\"},\"keyed\":false,\"min_doc_count\":1}}}}]",
              "start_time_in_millis" : 1589382266523,
              "running_time_in_nanos" : 1088083455,
              "cancellable" : true,
              "headers" : { }
            },
            "WKr4WGjVQTSKGJTVWHd0OA:23856285" : {
              "node" : "WKr4WGjVQTSKGJTVWHd0OA",
              "id" : 23856285,
              "type" : "transport",
              "action" : "indices:data/read/msearch",
              "description" : "",
              "start_time_in_millis" : 1589382266524,
              "running_time_in_nanos" : 1087027718,
              "cancellable" : false,
              "headers" : { }
            },
            "WKr4WGjVQTSKGJTVWHd0OA:23856286" : {
              "node" : "WKr4WGjVQTSKGJTVWHd0OA",
              "id" : 23856286,
              "type" : "transport",
              "action" : "indices:data/read/search",
              "description" : "indices[filebeat-hdfs-production-ilm-alias,filebeat-hdfs-production-2020.03.19-000002,filebeat-hdfs-production-2020.03.19-000003], types[], search_type[QUERY_THEN_FETCH], source[{\"size\":500,\"timeout\":\"30000ms\",\"query\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"from\":\"2020-04-12T23:00:00.000Z\",\"to\":\"2020-05-13T15:04:26.467Z\",\"include_lower\":true,\"include_upper\":true,\"format\":\"strict_date_optional_time\",\"boost\":1.0}}}],\"filter\":[{\"match_all\":{\"boost\":1.0}}],\"adjust_pure_negative\":true,\"boost\":1.0}},\"version\":true,\"_source\":{\"includes\":[],\"excludes\":[]},\"stored_fields\":\"*\",\"docvalue_fields\":[{\"field\":\"@timestamp\",\"format\":\"date_time\"},{\"field\":\"event.created\",\"format\":\"date_time\"},{\"field\":\"event.end\",\"format\":\"date_time\"},{\"field\":\"event.start\",\"format\":\"date_time\"},{\"field\":\"file.ctime\",\"format\":\"date_time\"},{\"field\":\"file.mtime\",\"format\":\"date_time\"}],\"script_fields\":{},\"sort\":[{\"@timestamp\":{\"order\":\"desc\",\"unmapped_type\":\"boolean\"}}],\"track_total_hits\":2147483647,\"aggregations\":{\"2\":{\"date_histogram\":{\"field\":\"@timestamp\",\"time_zone\":\"Europe/London\",\"fixed_interval\":\"12h\",\"offset\":0,\"order\":{\"_key\":\"asc\"},\"keyed\":false,\"min_doc_count\":1}}}}]",
              "start_time_in_millis" : 1589382266524,
              "running_time_in_nanos" : 1087035475,
              "cancellable" : true,
              "headers" : { }
            },
            "WKr4WGjVQTSKGJTVWHd0OA:23856283" : {
              "node" : "WKr4WGjVQTSKGJTVWHd0OA",
              "id" : 23856283,
              "type" : "transport",
              "action" : "indices:data/read/msearch",
              "description" : "",
              "start_time_in_millis" : 1589382266523,
              "running_time_in_nanos" : 1088159349,
              "cancellable" : false,
              "headers" : { }
            }
          }
        }
      }
    }

In the above example, WKr4WGjVQTSKGJTVWHd0OA:23856284 & WKr4WGjVQTSKGJTVWHd0OA:23856286 appear to be doing the same search. I see a similar story whenever i run queries from Kibana discover. Is this normal behaviour? If so why? If not then does anyone have any idea what might cause this?

Thanks

EDIT - I am running version 7.3 across the stack

Does anyone have any thoughts on this?

How many shards does your index have, and how many live on that node (primary or replica)?

Hi @warkolm,

In the example given I am searching multiple indices using a wildcard index pattern in Kibana filebeat-hdfs-production-* which for this pattern hits only two indices. These each have 3 primaries and 2 replicas spread over 6 nodes. (Hot / Warm) which are managed through ILM policies.

I see the same behaviour for other index-patterns where I have 100+ indices.

Thanks

I should also mention the searches are executed through Kibana which is installed on a coordinator-only node. I actually have 3 kibana instances running on 3 separate coordinator nodes behind HAProxy, for high availability. If that makes any difference

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.