App search - Filters - Arrays

Elastic Search
1

Hi there :wave:

I want to know what is the best filtering approach for my scenario. (Search API Filters)

Scenario:

Intro: I have arrays of values on the client side, and on the engine document schema one. I want documents to be returned only if there is an intersection between client side array and indexed document array.

Indexed document example:

    {
        id: "1",
        name: "name 1",
        author: "u1",
        idx_groups_a1: ["ga1", "ga2", "ga3", ..., "gaX000th"],
        idx_groups_b2: ["gb1", "gb2", "gb3", ..., "gbX000th"]
    }

On the client side let's say I have:

  • a1_groups = ["ga50", "ga51"]
  • a2_groups = ["gb1000", "gb1001"]

I want to return then: documents that match if there is an :

  • intersection between a1_groups and idx_groups_a1
    OR
  • intersection between a2_groups and idx_groups_b2.

I understand there are at least 2 possible ways of achieving, but I wanted to know which one is the preferred one in terms of performance, and in terms of limitations since a1_groups and a2_groups could have thousands of values.

Option A

This option is based of the requests that the App Search Generated Search UI performs:

    "filters": {
        "any": [
            {
                "any": [
                    {
                        "idx_groups_a1": "ga50"
                    },
                    {
                        "idx_groups_a1": "ga51"
                    }
                ]
            },
            {
                "any": [
                    {
                        "idx_groups_b2": "gb1000"
                    },
                    {
                        "idx_groups_b2": "gb1001"
                    }
                ]
            }
        ]
    },

Option B

    "filters": {
        "any": [
            {
                "idx_groups_a1": [
                    "ga50",
                    "ga51"
                ]
            },
            {
                "idx_groups_b2": [
                    "gb1000",
                    "gb1001"
                ]
            }
        ]
    }

Thanks in advance !

2

hello @Gerardo_Zenobi

Thank you for your detailed question!

Just so im clear about your example above, you want a filter that:

a document that has either:

  • "ga50" or "ga51" in idx_groups_a1
  • "gb1000" or "gb1001" in idx_groups_b2

If so your suggestions should be correct. Option B is the most optimal but depending on how many filter groups you require, you could hit the 32 max filter limit (Search API filters | Elastic App Search Documentation [8.4] | Elastic). With arrays you can have upto 1024 values before hitting the limit. Unfortunately there is no way to increase this limit AFAIK.

Hope this helps!

3

Hi @joemcelroy , thanks for the rapid response!

a document that has either:

  • "ga50" or "ga51" in idx_groups_a1
  • "gb1000" or "gb1001" in idx_groups_b2

Yes, your interpretation was correct: It is any match between 2 given arrays: as long as there is an intersection.length > 0, then the condition is true and the document should not be filtered out.

With arrays you can have upto 1024 values before hitting the limit.

With this you mean that within my filters in the query to EAP:

"idx_groups_a1": [ "v1", "v2","v3", ..., "v1024-MAX ! ! "]

A given value filter, stated in the array form like above, can't pass more than 1024 values ? Then if my use case needs to pass more than the limit, I would need to divide as follows ? :

(taking only idx_groups_a1 as example for the more than 1024 values case)

        "any": [
            {
                "idx_groups_a1":  [ "v1", "v2","v3", ..., "v1024-MAX ! ! "]
            },
            {
                "idx_groups_a1":  [ "v1024", "v1024","v1024", ..., "v2048-MAX ! ! "]
            },
            {
                "idx_groups_b2":  [ ...]
            },
            // ...
        ]

Is this legit ? (and possibly the only workaround provided we stay within the 32 max filters)

Thanks in advance.

4

Hey Gerardo!

Yes, you understood perfectly!

Yes! That is a good solution to mitigate the 1024 filter limit per array.

Thanks
Joe

1 Like
5

Thanks again @joemcelroy :+1:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.