[5.6.5] terminate_after and the Scroll API

[darin (Darin Douglass)]

All,

I'm seeing some peculiar behavior when using terminate_after in scrolls.

For local testing, I have a 1 node 5.6.5 cluster with indices configured with 1 shard:

{:26e1f492-59ca-459a-ab3f-5ed34ee125aa-0
 {:settings
  {:index
   {:number_of_replicas "0",
    :search
    {:slowlog
     {:threshold
      {:query
       {:warn "30s", :trace "10s", :debug "15s", :info "25s"}}}},
    :creation_date "1585178759881",
    :codec "best_compression",
    :mapper {:dynamic "false"},
    :analysis
    {:filter
     {:word_delim_with_orig
      {:type "word_delimiter", :preserve_original "true"},
      :possessive_stemmer
      {:name "possessive_english", :type "stemmer"}},
     :analyzer
     {:default
      {:filter ["standard" "lowercase" "possessive_stemmer" "kstem"],
       :tokenizer "classic"},
      :sortable {:filter ["lowercase"], :tokenizer "keyword"},
      :sonian_standard
      {:filter ["standard" "lowercase" "possessive_stemmer" "kstem"],
       :tokenizer "classic"},
      :email
      {:filter ["standard" "lowercase"], :tokenizer "uax_url_email"}}},
    :provided_name "26e1f492-59ca-459a-ab3f-5ed34ee125aa-0",
    :number_of_shards "1",
    :uuid "hHJhiISSQPWW9-rrQwtMRQ",
    :version {:created "5060599"}}}}}

I set up my test index with 10 documents that match the following query:

{:sort ["_doc"],
 :query
 {:bool
  {:must {:prefix {:body "foo"}}}},
 :_source #{"id"},
 :terminate_after 5,
 :size 2}

With a size = 2 and terminate_after = 5, I would expect to get 3 pages back:

• 2 2-hit pages and 1 1-hit page.
• On the final page, I would also expect terminated_early = true.

However, what I get back is actually:

• 5 2-hit pages, each with terminated_early = true and each with a total = 5
• 1 empty page signifying the scroll has fully "completed"

This behavior seems quite odd. Is my understanding of how size and terminate_after faulty? Does the scroll API honor terminate_after?

Thanks for the help!

Here are the results of the above scroll query, in order:

# --- page 1
{:_scroll_id
 "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAABUWeURLMDFDNmtSRXlOdHV5QzNTcTNNUQ==",
 :took 1,
 :timed_out false,
 :terminated_early true,
 :_shards {:total 1, :successful 1, :skipped 0, :failed 0},
 :hits
 {:total 5,
  :max_score nil,
  :hits
  [{:_index "26e1f492-59ca-459a-ab3f-5ed34ee125aa-0",
    :_type "msg",
    :_id "AAAAAAAAAAAAAAAAAAAAAA",
    :_score nil,
    :_source {:id "AAAAAAAAAAAAAAAAAAAAAA"},
    :sort [0]}
   {:_index "26e1f492-59ca-459a-ab3f-5ed34ee125aa-0",
    :_type "msg",
    :_id "AAAAAAAAAAEAAAAAAAAAAQ",
    :_score nil,
    :_source {:id "AAAAAAAAAAEAAAAAAAAAAQ"},
    :sort [1]}]}}

# --- page 2
{:_scroll_id
 "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAABUWeURLMDFDNmtSRXlOdHV5QzNTcTNNUQ==",
 :took 2,
 :timed_out false,
 :terminated_early true,
 :_shards {:total 1, :successful 1, :skipped 0, :failed 0},
 :hits
 {:total 5,
  :max_score nil,
  :hits
  [{:_index "26e1f492-59ca-459a-ab3f-5ed34ee125aa-0",
    :_type "msg",
    :_id "AAAAAAAAAAIAAAAAAAAAAg",
    :_score nil,
    :_source {:id "AAAAAAAAAAIAAAAAAAAAAg"},
    :sort [2]}
   {:_index "26e1f492-59ca-459a-ab3f-5ed34ee125aa-0",
    :_type "msg",
    :_id "AAAAAAAAAAMAAAAAAAAAAw",
    :_score nil,
    :_source {:id "AAAAAAAAAAMAAAAAAAAAAw"},
    :sort [3]}]}}

# --- page 3
{:_scroll_id
 "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAABUWeURLMDFDNmtSRXlOdHV5QzNTcTNNUQ==",
 :took 1,
 :timed_out false,
 :terminated_early true,
 :_shards {:total 1, :successful 1, :skipped 0, :failed 0},
 :hits
 {:total 5,
  :max_score nil,
  :hits
  [{:_index "26e1f492-59ca-459a-ab3f-5ed34ee125aa-0",
    :_type "msg",
    :_id "AAAAAAAAAAQAAAAAAAAABA",
    :_score nil,
    :_source {:id "AAAAAAAAAAQAAAAAAAAABA"},
    :sort [4]}
   {:_index "26e1f492-59ca-459a-ab3f-5ed34ee125aa-0",
    :_type "msg",
    :_id "AAAAAAAAAAUAAAAAAAAABQ",
    :_score nil,
    :_source {:id "AAAAAAAAAAUAAAAAAAAABQ"},
    :sort [5]}]}}

# --- page 4
{:_scroll_id
 "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAABUWeURLMDFDNmtSRXlOdHV5QzNTcTNNUQ==",
 :took 1,
 :timed_out false,
 :terminated_early true,
 :_shards {:total 1, :successful 1, :skipped 0, :failed 0},
 :hits
 {:total 5,
  :max_score nil,
  :hits
  [{:_index "26e1f492-59ca-459a-ab3f-5ed34ee125aa-0",
    :_type "msg",
    :_id "AAAAAAAAAAYAAAAAAAAABg",
    :_score nil,
    :_source {:id "AAAAAAAAAAYAAAAAAAAABg"},
    :sort [6]}
   {:_index "26e1f492-59ca-459a-ab3f-5ed34ee125aa-0",
    :_type "msg",
    :_id "AAAAAAAAAAcAAAAAAAAABw",
    :_score nil,
    :_source {:id "AAAAAAAAAAcAAAAAAAAABw"},
    :sort [7]}]}}

# --- page 5
{:_scroll_id
 "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAABUWeURLMDFDNmtSRXlOdHV5QzNTcTNNUQ==",
 :took 1,
 :timed_out false,
 :terminated_early true,
 :_shards {:total 1, :successful 1, :skipped 0, :failed 0},
 :hits
 {:total 5,
  :max_score nil,
  :hits
  [{:_index "26e1f492-59ca-459a-ab3f-5ed34ee125aa-0",
    :_type "msg",
    :_id "AAAAAAAAAAgAAAAAAAAACA",
    :_score nil,
    :_source {:id "AAAAAAAAAAgAAAAAAAAACA"},
    :sort [8]}
   {:_index "26e1f492-59ca-459a-ab3f-5ed34ee125aa-0",
    :_type "msg",
    :_id "AAAAAAAAAAkAAAAAAAAACQ",
    :_score nil,
    :_source {:id "AAAAAAAAAAkAAAAAAAAACQ"},
    :sort [9]}]}}

# --- page end
{:_scroll_id
 "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAABUWeURLMDFDNmtSRXlOdHV5QzNTcTNNUQ==",
 :took 1,
 :timed_out false,
 :terminated_early true,
 :_shards {:total 1, :successful 1, :skipped 0, :failed 0},
 :hits {:total 5, :max_score nil, :hits []}}