Documents including large arrays

Elastic Search
1

Hi,

I am using Elastic App Search, and I have a document that contains an array field that I am using to filter search requests.

e.g.

{
  name: "nameOfTheDoc"
  users: ["userA", "userB", "userC"]
}

In some rare document instances, the size of this array could exceed 50k elements.

Note that when I'm doing the search request, my filter field (in the request body) will only contain 1 element (which would need to be compared to the 50k elements in such a document).

Is this something that Elastic indexes and Elastic App Search can handle? I did not find a maximum array size in the documentation.

Thank you!

2

I don't think you'll want to manually create that type of filter for performance reasons, but it sounds like you may be interested in document level security. This will take care of user access for you. If you're using Connectors to ingest data into App Search, we have a guide to set them up. Hope that helps!

3

@Kathleen_DeRusso Thank you for your prompt reply and for suggesting document level security.

Unfortunately I'm in a case where permission cannot be grouped based on role or a similar concept.

You may see the document as a class, and some users were individually assigned to these classes. In Search, users should only see classes they are assigned to.

4

The guide that I linked in my original reply has examples that combine both groups and individual users.

50K is a LOT, if you can streamline that to be fewer items, you'll see much better performance.

5

Hi @Kathleen_DeRusso,

Thanks!

I understand that 50k is very large, however, due to the nature of our application, we need to handle these large arrays in some instances, at least for now. So we are trying to assess the impact it may have beforehand.

My understanding is that Elastic App Search can handle such arrays, but that performance will be impacted, perhaps significantly. Is that correct?

Note that only a small fraction of the documents would contain such a large array. In general, the size would be much smaller. Would you say that those few items could affect search performance overall? Do you have a rough idea of the impact it may have?

6

I can't really say one way or the other, you'll have to test with the size and shape of your data. Good luck!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.