# Is it possible to run text\_expansion query on a field that has an array of tokens generated by ELSER?

**URL:** <https://discuss.elastic.co/t/is-it-possible-to-run-text-expansion-query-on-a-field-that-has-an-array-of-tokens-generated-by-elser/350782>\
**Category:** Elastic Search\
**Tags:** elastic-app-search\
**Created:** [January 10, 2024, 6:25pm UTC](https://discuss.elastic.co/t/is-it-possible-to-run-text-expansion-query-on-a-field-that-has-an-array-of-tokens-generated-by-elser/350782 "2024-01-10T18:25:59Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![sneh3091](https://avatars.discourse-cdn.com/v4/letter/s/35a633/32.png) [@sneh3091](https://discuss.elastic.co/u/sneh3091)\
**Post date:** [January 10, 2024, 6:25pm UTC](https://discuss.elastic.co/t/is-it-possible-to-run-text-expansion-query-on-a-field-that-has-an-array-of-tokens-generated-by-elser/350782/1 "2024-01-10T18:25:59Z")

</div>

Hi we are using Elastic 8.10.4

We have a certain process that ingests documents( like PDF ) and runs a "script" processor to split text from those documents into multiple chunks and then runs an inference processor for ELSER on each of those chunks which then gets dumped into a single field "ml\_text" having an array of the tokens produced by ELSER like

```auto
...
"_source":{
"passages":[
 { "text": "abc"},
 { "text": "def"}
],
"ml_text": [
{"tokens": <tokens from ELSER on passages[0].text >, "model_id": ".elser_model_1"},
{"tokens": <tokens from ELSER on passages[1].text >, "model_id": ".elser_model_1"}
]
...

```

Question is , is it possible to run the text\_expansion on "ml\_text" for each of the chunks and return a matching chunk text from the "passages" array?

---

<div class="post-metadata">

**Author:** ![demjened](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/demjened/32/109159_2.png) [@demjened](https://discuss.elastic.co/u/demjened)\
**Post date:** [January 10, 2024, 8:46pm UTC](https://discuss.elastic.co/t/is-it-possible-to-run-text-expansion-query-on-a-field-that-has-an-array-of-tokens-generated-by-elser/350782/2 "2024-01-10T20:46:06Z")

</div>

Hello Sneha!

Yes this is possible. You need to set up your index to contain the chunks in a nested field, then run a text\_expansion query within a [nested query](https://www.elastic.co/guide/en/elasticsearch/reference/8.10/nested.html):

```auto
GET chunked_index/_search
{
  "query": {
    "nested": {
      "path": "ml_text",
      "query": { 
        "text_expansion": {
          "ml_text.tokens": {
            "model_id": ".elser_model_1",
            "model_text": "some text"
          }
        }
      },
      "inner_hits": {}
    }
  }
}

```

Note the `inner_hits` property which will ensure the matching nested chunks are returned along with documents, in the order of decreasing relevance per doc (and each having a relevance score). So within a returned document `inner_hits.ml_text.hits.hits[0]` will contain the most relevant chunk in that doc, `inner_hits.ml_text.hits.hits[1]` will be the 2nd most relevant and so on.

In order to correlate the chunks with the actual passages you can follow either of these two strategies:

1. Embed the passage text in the chunks, this way it's directly accessible from `inner_hits`;
2. Postprocess the documents: for each doc, get the ordinal of the top chunk (inner\_hits.ml\_text.hits.hits[0].offset`) and look up the passage from the `passages` array of the same doc.

Hope this helps!

---

<div class="post-metadata">

**Author:** ![sneh3091](https://avatars.discourse-cdn.com/v4/letter/s/35a633/32.png) [@sneh3091](https://discuss.elastic.co/u/sneh3091)\
**Post date:** [January 10, 2024, 9:28pm UTC](https://discuss.elastic.co/t/is-it-possible-to-run-text-expansion-query-on-a-field-that-has-an-array-of-tokens-generated-by-elser/350782/4 "2024-01-10T21:28:56Z")

</div>

Hi Adam,

Thank you so much for your help!

`Re: 1. Embed the passage text in the chunks, this way it's directly accessible from inner_hits;`

Do you mean that instead of storing the texts in "passage", we store the text field within each object of ml\_text?

---

<div class="post-metadata">

**Author:** ![demjened](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/demjened/32/109159_2.png) [@demjened](https://discuss.elastic.co/u/demjened)\
**Post date:** [January 10, 2024, 9:47pm UTC](https://discuss.elastic.co/t/is-it-possible-to-run-text-expansion-query-on-a-field-that-has-an-array-of-tokens-generated-by-elser/350782/5 "2024-01-10T21:47:09Z")

</div>

Yes, correct - store each passage text right next to the corresponding vector embedding.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 7, 2024, 9:47pm UTC](https://discuss.elastic.co/t/is-it-possible-to-run-text-expansion-query-on-a-field-that-has-an-array-of-tokens-generated-by-elser/350782/6 "2024-02-07T21:47:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.