"The heap is empty" error on KNN search

abay.kenzhekhan · December 19, 2022, 10:58am

I am doing following KNN search request

{
	"filter": {
		"bool": {
			"must_not": {
				"term": {
					"some_field": 14642652
				}
			},
			"must": {
				"term": {
					"some_field2": true
				}
			}
		}
	},
	"knn": {
		"field": "photo_vector",
		"k": 5,
		"num_candidates": 5,
		"query_vector": [
			...
		]
	}
}

ES responses with 200 code, but I get following error on shards field:

{
	"took": 522,
	"timed_out": false,
	"_shards": {
		"total": 12,
		"successful": 2,
		"skipped": 0,
		"failed": 10,
		"failures": [
			{
				"shard": 0,
				"index": "photos-v4",
				"node": "ZF73KImLQ2qVcRhedvBKcA",
				"reason": {
					"type": "illegal_state_exception",
					"reason": "The heap is empty"
				}
			}
		]
	},
       "hits": {...}
}

If I remove condition ""some_field2": true", everything is ok.
What can be the reason for this error?

BenTrent · December 19, 2022, 2:16pm

@abay.kenzhekhan

My gut tells me that this is because the query filter in KNN is very restrictive, so for certain document segments, no documents are found. But the The heap is empty error is thrown by an internal structure that is used many different places.

To be sure, can you re-run your failing query with error_trace=true so we get the full Java stacktrace to help debug the issue?

abay.kenzhekhan · December 20, 2022, 4:42am

Unfortunately, it returns the same response with no additional info. What else can you suggest to do? Thanks

BenTrent · December 20, 2022, 5:48pm

Ah, please set allow_partial_search_results=false and error_trace=true That may work.

We have tests with restrictive filters (with empty results), so something strange indeed is going on.

Also, your Elasticsearch version would help as well

abay.kenzhekhan · December 21, 2022, 3:27am

Elastic version: 8.3.1

Here is response:

{
	"error": {
		"root_cause": [
			{
				"type": "illegal_argument_exception",
				"reason": "request [/photos-alias/_knn_search] contains unrecognized parameter: [allow_partial_search_results]",
				"stack_trace": "org.elasticsearch.ElasticsearchException$1: request [/photos-alias/_knn_search] contains unrecognized parameter: [allow_partial_search_results]\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:638)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:566)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:145)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:101)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:81)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:391)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:468)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:304)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:371)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:450)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:345)\n\tat org.elasticsearch.transport.netty4@8.3.1/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:98)\n\tat org.elasticsearch.transport.netty4@8.3.1/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:88)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.handler@4.1.76.Final/io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)\n\tat io.netty.common@4.1.76.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)\n\tat io.netty.common@4.1.76.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: java.lang.IllegalArgumentException: request [/photos-alias/_knn_search] contains unrecognized parameter: [allow_partial_search_results]\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:94)\n\tat org.elasticsearch.security@8.3.1/org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:119)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:389)\n\t... 48 more\n"
			}
		],
		"type": "illegal_argument_exception",
		"reason": "request [/photos-alias/_knn_search] contains unrecognized parameter: [allow_partial_search_results]",
		"stack_trace": "java.lang.IllegalArgumentException: request [/photos-alias/_knn_search] contains unrecognized parameter: [allow_partial_search_results]\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:94)\n\tat org.elasticsearch.security@8.3.1/org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:119)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:389)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:468)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:304)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:371)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:450)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:345)\n\tat org.elasticsearch.transport.netty4@8.3.1/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:98)\n\tat org.elasticsearch.transport.netty4@8.3.1/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:88)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.handler@4.1.76.Final/io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)\n\tat io.netty.common@4.1.76.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)\n\tat io.netty.common@4.1.76.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"
	},
	"status": 400
}

It seems elastic doesn't understand allow_partial_search_results. Also what I found is that there is no error if instead of term condition with bool, I use some integer condition, everything works. It seems there is some problem with shards and bool condition.

abay.kenzhekhan · December 21, 2022, 4:23am

I have some another question. We have to approach to filter KNN result:

Using filter. For instance, filter:{"term": {"n": 1}}
Using query. For instance, query:{"term": {"n": 1}}

The difference between them is that query with knn works like a OR. But we can anyway use it like a simple filter using min_score. The question is which on above of approaches is more suitable for filtering and have best speed? It seems filter is applied only after knn search gets completed, while query applied during (but why it works like a OR?)

BenTrent · December 21, 2022, 3:40pm

If you only want to filter documents with KNN, you should supply that filter within the KNN object.

POST <index_name>/_search?allow_partial_search_results=false&error_trace=true should definitely work. I have just verified that these query parameters work just fine.

The main thing here, is if you want the term query to provide scores to the document, or you only want documents scored according to KNN.

Here is what a filter looks like with using KNN (just now noticed you are using a filter in the top level query my bad...)

{
  "knn": {
    "filter": {
      "bool": {
        "must_not": {
          "term": {
            "some_field": 14642652
          }
        },
        "must": {
          "term": {
            "some_field2": true
          }
        }
      }
    },
    "field": "photo_vector",
    "k": 5,
    "num_candidates": 5,
    "query_vector": [...]
  }
}

^ This is the way to filter with KNN search. This way only vectors that match the filter are considered.

If you want to use "hybrid search" (meaning docs are scored according to term query & KNN and the disjoint OR is returned), then use a term query clause in addition to the knn clause.

Example from the docs

{
  "query": {
    "match": {
      "title": {
        "query": "mountain lake",
        "boost": 0.9
      }
    }
  },
  "knn": {
    "field": "image-vector",
    "query_vector": [54, 10, -2],
    "k": 5,
    "num_candidates": 50,
    "boost": 0.1
  },
  "size": 10
}

abay.kenzhekhan · December 22, 2022, 11:09am

Thanks for the great answer. But it leads to a lot of questions:

filter might be inside knn and outside. Are there any difference? I hope they work the same way.
is filter in knn the same as post_filter? Does ES do KNN first and only after do what is inside filter?
query with knn might be used just like filter. Even if it works like OR, we can just use min_score or just take some top scored documents. It creates a question like which one is faster and performs better: filter or query?

BenTrent · December 23, 2022, 7:48pm

Yes, there is a difference, doing a filter outside KNN is an OR combination. Either docs match the filter, or KNN or both. Note, the filtered documents are unscored, but they may be scored by the KNN query.
It is not a post-filter, but it is an AND combination.
Yes, this is how you should do it.

Something like this would be an OR combination with a filter & KNN.

{
       "query": {
           "bool": {
                "must_not": { "term": {"some_field": 14642652} },
                "filter": {"term": {"some_field2": true}}
           }
        },
	"knn": {
		"field": "photo_vector",
		"k": 5,
		"num_candidates": 5,
		"query_vector": [
			...
		]
	}
}

To get the full stack trace to fully debug your original issue, I have confirmed the query parameters allow_partial_search_results=false&error_trace=true are what you need. If you want to dig into that further.

abay.kenzhekhan · December 26, 2022, 3:37am

There is a great article telling that it is not possible to do filtering first and only after, on small subset of data, making the KNN search. According to this article, we can say that there is no benefit of adding some filter, because anyway, KNN does his job first (because of graphs) and only after (or during), docs will be filtered. The performance would be way better if filtering does his job first and only after, KNN search will be executed on smaller data. Are my observations are correct?

BenTrent · December 26, 2022, 12:28pm

We don't do pre/post filtering. We take advantage of the KNN index and the filter index living in the same document. Additionally, an optimization kicks in if the filter set is tiny, we skip running through the HNSW graph as it wouldn't provide benefit.

Doing an 'OR' combination does not reduce the work done by KNN at all. It is just saying, either it's a closest neighbor or it's part of this filter.

If you want to do a filter that says "I only want these documents and give me the ones that are closest neighbors" the example I gave you is what you need

system · January 23, 2023, 12:28pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.