Why do I see multiple search tasks for a single search in Kibana

Elastic Stack Kibana
1

As the title suggests when I run a query in Kibana using the discover app I see two identical search actions using the task management API. Just curious why this is and if it is actually running the same query twice against my ES cluster

Example:
Discover > filebeat-hdfs-production-* Index Pattern (All HDFS logs) > Last 30 days (Rounded)

Tasks management API

    _tasks?actions=*search&pretty&detailed"

    {
      "nodes" : {
        "WKr4WGjVQTSKGJTVWHd0OA" : {
          "name" : "<redacted>",
          "transport_address" : "<redacted>:9300",
          "host" : "<redacted>",
          "ip" : "<redacted>:9300",
          "roles" : [ ],
          "attributes" : {
            "xpack.installed" : "true"
          },
          "tasks" : {
            "WKr4WGjVQTSKGJTVWHd0OA:23856284" : {
              "node" : "WKr4WGjVQTSKGJTVWHd0OA",
              "id" : 23856284,
              "type" : "transport",
              "action" : "indices:data/read/search",
              "description" : "indices[filebeat-hdfs-production-ilm-alias,filebeat-hdfs-production-2020.03.19-000002,filebeat-hdfs-production-2020.03.19-000003], types[], search_type[QUERY_THEN_FETCH], source[{\"size\":500,\"timeout\":\"30000ms\",\"query\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"from\":\"2020-04-12T23:00:00.000Z\",\"to\":\"2020-05-13T15:04:26.467Z\",\"include_lower\":true,\"include_upper\":true,\"format\":\"strict_date_optional_time\",\"boost\":1.0}}}],\"filter\":[{\"match_all\":{\"boost\":1.0}}],\"adjust_pure_negative\":true,\"boost\":1.0}},\"version\":true,\"_source\":{\"includes\":[],\"excludes\":[]},\"stored_fields\":\"*\",\"docvalue_fields\":[{\"field\":\"@timestamp\",\"format\":\"date_time\"},{\"field\":\"event.created\",\"format\":\"date_time\"},{\"field\":\"event.end\",\"format\":\"date_time\"},{\"field\":\"event.start\",\"format\":\"date_time\"},{\"field\":\"file.ctime\",\"format\":\"date_time\"},{\"field\":\"file.mtime\",\"format\":\"date_time\"}],\"script_fields\":{},\"sort\":[{\"@timestamp\":{\"order\":\"desc\",\"unmapped_type\":\"boolean\"}}],\"track_total_hits\":2147483647,\"aggregations\":{\"2\":{\"date_histogram\":{\"field\":\"@timestamp\",\"time_zone\":\"Europe/London\",\"fixed_interval\":\"12h\",\"offset\":0,\"order\":{\"_key\":\"asc\"},\"keyed\":false,\"min_doc_count\":1}}}}]",
              "start_time_in_millis" : 1589382266523,
              "running_time_in_nanos" : 1088083455,
              "cancellable" : true,
              "headers" : { }
            },
            "WKr4WGjVQTSKGJTVWHd0OA:23856285" : {
              "node" : "WKr4WGjVQTSKGJTVWHd0OA",
              "id" : 23856285,
              "type" : "transport",
              "action" : "indices:data/read/msearch",
              "description" : "",
              "start_time_in_millis" : 1589382266524,
              "running_time_in_nanos" : 1087027718,
              "cancellable" : false,
              "headers" : { }
            },
            "WKr4WGjVQTSKGJTVWHd0OA:23856286" : {
              "node" : "WKr4WGjVQTSKGJTVWHd0OA",
              "id" : 23856286,
              "type" : "transport",
              "action" : "indices:data/read/search",
              "description" : "indices[filebeat-hdfs-production-ilm-alias,filebeat-hdfs-production-2020.03.19-000002,filebeat-hdfs-production-2020.03.19-000003], types[], search_type[QUERY_THEN_FETCH], source[{\"size\":500,\"timeout\":\"30000ms\",\"query\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"from\":\"2020-04-12T23:00:00.000Z\",\"to\":\"2020-05-13T15:04:26.467Z\",\"include_lower\":true,\"include_upper\":true,\"format\":\"strict_date_optional_time\",\"boost\":1.0}}}],\"filter\":[{\"match_all\":{\"boost\":1.0}}],\"adjust_pure_negative\":true,\"boost\":1.0}},\"version\":true,\"_source\":{\"includes\":[],\"excludes\":[]},\"stored_fields\":\"*\",\"docvalue_fields\":[{\"field\":\"@timestamp\",\"format\":\"date_time\"},{\"field\":\"event.created\",\"format\":\"date_time\"},{\"field\":\"event.end\",\"format\":\"date_time\"},{\"field\":\"event.start\",\"format\":\"date_time\"},{\"field\":\"file.ctime\",\"format\":\"date_time\"},{\"field\":\"file.mtime\",\"format\":\"date_time\"}],\"script_fields\":{},\"sort\":[{\"@timestamp\":{\"order\":\"desc\",\"unmapped_type\":\"boolean\"}}],\"track_total_hits\":2147483647,\"aggregations\":{\"2\":{\"date_histogram\":{\"field\":\"@timestamp\",\"time_zone\":\"Europe/London\",\"fixed_interval\":\"12h\",\"offset\":0,\"order\":{\"_key\":\"asc\"},\"keyed\":false,\"min_doc_count\":1}}}}]",
              "start_time_in_millis" : 1589382266524,
              "running_time_in_nanos" : 1087035475,
              "cancellable" : true,
              "headers" : { }
            },
            "WKr4WGjVQTSKGJTVWHd0OA:23856283" : {
              "node" : "WKr4WGjVQTSKGJTVWHd0OA",
              "id" : 23856283,
              "type" : "transport",
              "action" : "indices:data/read/msearch",
              "description" : "",
              "start_time_in_millis" : 1589382266523,
              "running_time_in_nanos" : 1088159349,
              "cancellable" : false,
              "headers" : { }
            }
          }
        }
      }
    }

In the above example, WKr4WGjVQTSKGJTVWHd0OA:23856284 & WKr4WGjVQTSKGJTVWHd0OA:23856286 appear to be doing the same search. I see a similar story whenever i run queries from Kibana discover. Is this normal behaviour? If so why? If not then does anyone have any idea what might cause this?

Thanks

EDIT - I am running version 7.3 across the stack

2

Does anyone have any thoughts on this?

3

How many shards does your index have, and how many live on that node (primary or replica)?

4

Hi @warkolm,

In the example given I am searching multiple indices using a wildcard index pattern in Kibana filebeat-hdfs-production-* which for this pattern hits only two indices. These each have 3 primaries and 2 replicas spread over 6 nodes. (Hot / Warm) which are managed through ILM policies.

I see the same behaviour for other index-patterns where I have 100+ indices.

Thanks

5

I should also mention the searches are executed through Kibana which is installed on a coordinator-only node. I actually have 3 kibana instances running on 3 separate coordinator nodes behind HAProxy, for high availability. If that makes any difference

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.