# Documents including large arrays

**URL:** <https://discuss.elastic.co/t/documents-including-large-arrays/363125>\
**Category:** Elastic Search\
**Created:** [July 15, 2024, 8:06am UTC](https://discuss.elastic.co/t/documents-including-large-arrays/363125 "2024-07-15T08:06:38Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![ismaelalt](https://avatars.discourse-cdn.com/v4/letter/i/aeb1de/32.png) [@ismaelalt](https://discuss.elastic.co/u/ismaelalt)\
**Post date:** [July 15, 2024, 8:06am UTC](https://discuss.elastic.co/t/documents-including-large-arrays/363125/1 "2024-07-15T08:06:38Z")

</div>

Hi,

I am using Elastic App Search, and I have a document that contains an array field that I am using to filter search requests.

e.g.

```auto
{
  name: "nameOfTheDoc"
  users: ["userA", "userB", "userC"]
}

```

In some rare document instances, the size of this array could exceed 50k elements.

Note that when I'm doing the search request, my filter field (in the request body) will only contain 1 element (which would need to be compared to the 50k elements in such a document).

Is this something that Elastic indexes and Elastic App Search can handle? I did not find a maximum array size in the documentation.

Thank you!

---

<div class="post-metadata">

**Author:** ![Kathleen\_DeRusso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kathleen_derusso/32/132039_2.png) [@Kathleen\_DeRusso](https://discuss.elastic.co/u/Kathleen_DeRusso)\
**Post date:** [July 15, 2024, 12:07pm UTC](https://discuss.elastic.co/t/documents-including-large-arrays/363125/2 "2024-07-15T12:07:26Z")

</div>

I don't think you'll want to manually create that type of filter for performance reasons, but it sounds like you may be interested in [document level security](https://www.elastic.co/guide/en/elasticsearch/reference/current/document-level-security.html). This will take care of user access for you. If you're using Connectors to ingest data into App Search, we have a [guide](https://www.elastic.co/guide/en/enterprise-search/current/dls-connectors-app-search.html) to set them up. Hope that helps!

---

<div class="post-metadata">

**Author:** ![ismaelalt](https://avatars.discourse-cdn.com/v4/letter/i/aeb1de/32.png) [@ismaelalt](https://discuss.elastic.co/u/ismaelalt)\
**Post date:** [July 15, 2024, 4:06pm UTC](https://discuss.elastic.co/t/documents-including-large-arrays/363125/3 "2024-07-15T16:06:13Z")

</div>

@Kathleen_DeRusso Thank you for your prompt reply and for suggesting document level security.

Unfortunately I'm in a case where permission cannot be grouped based on role or a similar concept.

You may see the document as a class, and some users were individually assigned to these classes. In Search, users should only see classes they are assigned to.

---

<div class="post-metadata">

**Author:** ![Kathleen\_DeRusso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kathleen_derusso/32/132039_2.png) [@Kathleen\_DeRusso](https://discuss.elastic.co/u/Kathleen_DeRusso)\
**Post date:** [July 15, 2024, 5:26pm UTC](https://discuss.elastic.co/t/documents-including-large-arrays/363125/4 "2024-07-15T17:26:32Z")

</div>

The [guide](https://www.elastic.co/guide/en/enterprise-search/current/dls-connectors-app-search.html) that I linked in my original reply has examples that combine both groups and individual users.

50K is a LOT, if you can streamline that to be fewer items, you'll see much better performance.

---

<div class="post-metadata">

**Author:** ![ismaelalt](https://avatars.discourse-cdn.com/v4/letter/i/aeb1de/32.png) [@ismaelalt](https://discuss.elastic.co/u/ismaelalt)\
**Post date:** [July 16, 2024, 8:39am UTC](https://discuss.elastic.co/t/documents-including-large-arrays/363125/5 "2024-07-16T08:39:41Z")

</div>

Hi @Kathleen_DeRusso,

Thanks!

I understand that 50k is very large, however, due to the nature of our application, we need to handle these large arrays in some instances, at least for now. So we are trying to assess the impact it may have beforehand.

My understanding is that Elastic App Search can handle such arrays, but that performance will be impacted, perhaps significantly. Is that correct?

Note that only a small fraction of the documents would contain such a large array. In general, the size would be much smaller. Would you say that those few items could affect search performance overall? Do you have a rough idea of the impact it may have?

---

<div class="post-metadata">

**Author:** ![Kathleen\_DeRusso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kathleen_derusso/32/132039_2.png) [@Kathleen\_DeRusso](https://discuss.elastic.co/u/Kathleen_DeRusso)\
**Post date:** [July 16, 2024, 12:15pm UTC](https://discuss.elastic.co/t/documents-including-large-arrays/363125/6 "2024-07-16T12:15:23Z")

</div>

I can't really say one way or the other, you'll have to test with the size and shape of your data. Good luck!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 13, 2024, 12:15pm UTC](https://discuss.elastic.co/t/documents-including-large-arrays/363125/7 "2024-08-13T12:15:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.