I am doing following KNN search request
{
"filter": {
"bool": {
"must_not": {
"term": {
"some_field": 14642652
}
},
"must": {
"term": {
"some_field2": true
}
}
}
},
"knn": {
"field": "photo_vector",
"k": 5,
"num_candidates": 5,
"query_vector": [
...
]
}
}
ES responses with 200 code, but I get following error on shards field:
{
"took": 522,
"timed_out": false,
"_shards": {
"total": 12,
"successful": 2,
"skipped": 0,
"failed": 10,
"failures": [
{
"shard": 0,
"index": "photos-v4",
"node": "ZF73KImLQ2qVcRhedvBKcA",
"reason": {
"type": "illegal_state_exception",
"reason": "The heap is empty"
}
}
]
},
"hits": {...}
}
If I remove condition ""some_field2": true", everything is ok.
BenTrent
December 19, 2022, 2:16pm
2
@abay.kenzhekhan
My gut tells me that this is because the query filter in KNN is very restrictive, so for certain document segments, no documents are found. But the The heap is empty error is thrown by an internal structure that is used many different places.
To be sure, can you re-run your failing query with error_trace=true so we get the full Java stacktrace to help debug the issue?
1 Like
Unfortunately, it returns the same response with no additional info. What else can you suggest to do? Thanks
BenTrent
December 20, 2022, 5:48pm
4
Ah, please set allow_partial_search_results=false and error_trace=true That may work.
We have tests with restrictive filters (with empty results), so something strange indeed is going on.
Also, your Elasticsearch version would help as well
1 Like
Elastic version: 8.3.1
Here is response:
{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "request [/photos-alias/_knn_search] contains unrecognized parameter: [allow_partial_search_results]",
"stack_trace": "org.elasticsearch.ElasticsearchException$1: request [/photos-alias/_knn_search] contains unrecognized parameter: [allow_partial_search_results]\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:638)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:566)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:145)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:101)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:81)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:391)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:468)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:304)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:371)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:450)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:345)\n\tat org.elasticsearch.transport.netty4@8.3.1/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:98)\n\tat org.elasticsearch.transport.netty4@8.3.1/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:88)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.handler@4.1.76.Final/io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)\n\tat io.netty.common@4.1.76.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)\n\tat io.netty.common@4.1.76.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: java.lang.IllegalArgumentException: request [/photos-alias/_knn_search] contains unrecognized parameter: [allow_partial_search_results]\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:94)\n\tat org.elasticsearch.security@8.3.1/org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:119)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:389)\n\t... 48 more\n"
}
],
"type": "illegal_argument_exception",
"reason": "request [/photos-alias/_knn_search] contains unrecognized parameter: [allow_partial_search_results]",
"stack_trace": "java.lang.IllegalArgumentException: request [/photos-alias/_knn_search] contains unrecognized parameter: [allow_partial_search_results]\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:94)\n\tat org.elasticsearch.security@8.3.1/org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:119)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:389)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:468)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:304)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:371)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:450)\n\tat org.elasticsearch.server@8.3.1/org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:345)\n\tat org.elasticsearch.transport.netty4@8.3.1/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:98)\n\tat org.elasticsearch.transport.netty4@8.3.1/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:88)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.handler@4.1.76.Final/io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.codec@4.1.76.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586)\n\tat io.netty.transport@4.1.76.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)\n\tat io.netty.common@4.1.76.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)\n\tat io.netty.common@4.1.76.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"
},
"status": 400
}
It seems elastic doesn't understand allow_partial_search_results. Also what I found is that there is no error if instead of term condition with bool, I use some integer condition, everything works. It seems there is some problem with shards and bool condition.
I have some another question. We have to approach to filter KNN result:
Using filter. For instance, filter:{"term": {"n": 1}}
Using query. For instance, query:{"term": {"n": 1}}
The difference between them is that query with knn works like a OR. But we can anyway use it like a simple filter using min_score. The question is which on above of approaches is more suitable for filtering and have best speed? It seems filter is applied only after knn search gets completed, while query applied during (but why it works like a OR?)
BenTrent
December 21, 2022, 3:40pm
7
If you only want to filter documents with KNN, you should supply that filter within the KNN object.
POST <index_name>/_search?allow_partial_search_results=false&error_trace=true should definitely work. I have just verified that these query parameters work just fine.
The main thing here, is if you want the term query to provide scores to the document, or you only want documents scored according to KNN.
Here is what a filter looks like with using KNN (just now noticed you are using a filter in the top level query
{
"knn": {
"filter": {
"bool": {
"must_not": {
"term": {
"some_field": 14642652
}
},
"must": {
"term": {
"some_field2": true
}
}
}
},
"field": "photo_vector",
"k": 5,
"num_candidates": 5,
"query_vector": [...]
}
}
^ This is the way to filter with KNN search. This way only vectors that match the filter are considered.
If you want to use "hybrid search" (meaning docs are scored according to term query & KNN and the disjoint OR is returned), then use a term query clause in addition to the knn clause.
Example from the docs
{
"query": {
"match": {
"title": {
"query": "mountain lake",
"boost": 0.9
}
}
},
"knn": {
"field": "image-vector",
"query_vector": [54, 10, -2],
"k": 5,
"num_candidates": 50,
"boost": 0.1
},
"size": 10
}
1 Like
Thanks for the great answer. But it leads to a lot of questions:
filter might be inside knn and outside. Are there any difference? I hope they work the same way.
is filter in knn the same as post_filter? Does ES do KNN first and only after do what is inside filter?
query with knn might be used just like filter. Even if it works like OR, we can just use min_score or just take some top scored documents. It creates a question like which one is faster and performs better: filter or query?
BenTrent
December 23, 2022, 7:48pm
9
Yes, there is a difference, doing a filter outside KNN is an OR combination. Either docs match the filter, or KNN or both. Note, the filtered documents are unscored, but they may be scored by the KNN query.
It is not a post-filter, but it is an AND combination.
Yes, this is how you should do it.
Something like this would be an OR combination with a filter & KNN.
{
"query": {
"bool": {
"must_not": { "term": {"some_field": 14642652} },
"filter": {"term": {"some_field2": true}}
}
},
"knn": {
"field": "photo_vector",
"k": 5,
"num_candidates": 5,
"query_vector": [
...
]
}
}
To get the full stack trace to fully debug your original issue, I have confirmed the query parameters allow_partial_search_results=false&error_trace=true are what you need. If you want to dig into that further.
1 Like
BenTrent
December 26, 2022, 12:28pm
11
We don't do pre/post filtering. We take advantage of the KNN index and the filter index living in the same document. Additionally, an optimization kicks in if the filter set is tiny, we skip running through the HNSW graph as it wouldn't provide benefit.
Doing an 'OR' combination does not reduce the work done by KNN at all. It is just saying, either it's a closest neighbor or it's part of this filter.
If you want to do a filter that says "I only want these documents and give me the ones that are closest neighbors" the example I gave you is what you need
1 Like
system
January 23, 2023, 12:28pm
12
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
my bad...)