Filter documents based on their Guid[] array property to only retrieve those containing all the values from an incoming Guid[] array or more

Elastic Stack Elasticsearch
1

Hello,

I'm using .NET 6.0 and Elastic.Clients.Elasticsearch V8.15.10.

I have a title index with a property Guid[] Genres that is an array of Guids.
In my incoming query I expect the same, an array of Guids (query.Genres).

I'm trying to filter out the titles index to only retrieve those that have all the incoming Guids inside their Genres property or more, in other words the title document is a hit, if it's Genres contains everything in the query.Genres or more.

A simple example without using Guids to simplify the idea:

query.Genres: {0, 1}

titleDoc1.Genres: {0,1,2}
titleDoc2.Genres: {0,2}

Result: titleDoc1

My TitleDocument:


public class TitleDocument {
    .
    .
    .
    public Guid[] Cast { get; set; }
}

My Query:


        var result = await _client.SearchAsync<TitleDocument>(req =>
        {
            req.Index(_indices.Value.Titles)
            .Query(q => q

            );

        });

I'm not sure exactly how to implement the filtering, and I couldn't find any documentation or resources that visits this issue with the 8.15 version. I'm thankful for any help you can provide.

Thank you.

2

I am not a .NET developer so do not know that client. It does however sound like you want to use the terms set query.

3

Thank you for your response.

I've read about it and you're right, it's exactly what I'm looking for, thank you.

However, I've run into a new issue when I try to execute my query.

This is currently how my Query looks like:

var result = await _client.SearchAsync<TitleDocument>(req =>
{
    req.Index(_indices.Value.Titles)
    .Query(q => q
        .Bool(b =>
        {
            // Handle Cast filtering
            b.Filter(f => f
                .TermsSet(t => t
                    .Field(t => t.Cast)
                    .MinimumShouldMatch("required_matches")
                    .Terms(query.Cast.Select(id => (FieldValue)id.ToString()).ToList())
                )
            );
        })
    );
});

When I run the code and try it out, it throws an exception. Logs in the Microsoft Visual Studio Debug Console:

[13:57:31 ERR] Failed to process query SearchTitleQuery with error: 
Invalid Elasticsearch response built from an unsuccessful (400) low level call on POST: /manasa-vod-titles/_search
Exception: Request failed to execute. 
Call: Status code 400 from: POST /manasa-vod-titles/_search. 
ServerError: 
Type: x_content_parse_exception 
Reason: "[1:73] [bool] failed to parse field [filter]" 
CausedBy: 
    Type: parsing_exception 
    Reason: "[terms_set] query does not support [minimum_should_match]"

# Audit trail of this API call:
- [1] BadResponse: Node: http://192.168.0.100:9200/ 
  Took: 00:00:00.1260109

# OriginalException: 
Elastic.Transport.TransportException: 
Request failed to execute. 
Call: Status code 400 from: POST /manasa-vod-titles/_search. 
ServerError: 
Type: x_content_parse_exception 
Reason: "[1:73] [bool] failed to parse field [filter]" 
CausedBy: 
    Type: parsing_exception 
    Reason: "[terms_set] query does not support [minimum_should_match]"

# Request: 
<Request stream not captured or already read to completion by serializer. 
Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>

# Response: 
{
    "error": {
        "root_cause": [
            {
                "type": "parsing_exception",
                "reason": "[terms_set] query does not support [minimum_should_match]",
                "line": 1,
                "col": 73
            }
        ],
        "type": "x_content_parse_exception",
        "reason": "[1:73] [bool] failed to parse field [filter]",
        "caused_by": {
            "type": "parsing_exception",
            "reason": "[terms_set] query does not support [minimum_should_match]",
            "line": 1,

I get the same thing when using .MinimumShouldMatch(2) (integer) instead of MinimumShouldMatch("required_matches") (string.)

If I try to remove it, this the query looks like this:

var result = await _client.SearchAsync<TitleDocument>(req =>
{
    req.Index(_indices.Value.Titles)
    .Query(q => q
        .Bool(b =>
        {
            // Handle Cast filtering
            b.Filter(f => f
                .TermsSet(t => t
                    .Field(t => t.Cast)
                    .Terms(query.Cast.Select(id => (FieldValue)id.ToString()).ToList())
                )
            );
        })
    );
});

I get this error in the same debug console:

[14:11:32 ERR] Failed to process query SearchTitleQuery with error 
Invalid Elasticsearch response built from an unsuccessful (400) low level call on POST: /manasa-vod-titles/_search
Exception: Request failed to execute. 
Call: Status code 400 from: POST /manasa-vod-titles/_search. 
ServerError: 
Type: search_phase_execution_exception 
Reason: "all shards failed"

# Audit trail of this API call: 
- [1] BadResponse: Node: http://192.168.0.100:9200/ 
  Took: 00:00:00.1269650

# OriginalException: 
Elastic.Transport.TransportException: 
Request failed to execute. 
Call: Status code 400 from: POST /manasa-vod-titles/_search. 
ServerError: 
Type: search_phase_execution_exception 
Reason: "all shards failed"

# Request: 
<Request stream not captured or already read to completion by serializer. 
Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>

# Response: 
{
    "error": {
        "root_cause": [
            {
                "type": "query_shard_exception",
                "reason": "failed to create query: No minimum should match has been specified",
                "index_uuid": "f1_veTjSShKNAhQC969CKg",
                "index": "manasa-vod-titles"
            }
        ],
        "type": "search_phase_execution_exception",
        "reason": "all shards failed",
        "phase": "query",
        "grouped": true,
        "failed_shards": [
            {
                "shard": 0,
                "index": "manasa-vod-titles",
                "node": "AQ--wA-fQOiWVZPuoW8Iwg",
                "reason": {
                    "type": "query_shard_exception",
                    "reason": "failed to create query: No minimum should match has been specified",
                    "index_uuid": "f1_veTjSShKNAhQC969CKg",
                    "index": "manasa-vod-titles",
                    "caused_by": {
                        "type": "illegal_state_exception",
                        "reason": "No minimum should match has been specified"
                    }
                }
            }
        ]
    },
    "status": 400
}

Reading the terms set query I'm assuming I perhaps have to add a minimum field inside the TitleDocument itself, and with every new document put in the database, but I'm not sure if that's the case, and if it was, I'm not sure how to do that, or how would that look like...

Thank you