# Unable to perform hybrid search

**URL:** <https://discuss.elastic.co/t/unable-to-perform-hybrid-search/372393>\
**Category:** Elastic Search\
**Created:** [December 26, 2024, 4:41am UTC](https://discuss.elastic.co/t/unable-to-perform-hybrid-search/372393 "2024-12-26T04:41:33Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![harish32](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harish32/32/137750_2.png) [@harish32](https://discuss.elastic.co/u/harish32)\
**Post date:** [December 26, 2024, 4:41am UTC](https://discuss.elastic.co/t/unable-to-perform-hybrid-search/372393/1 "2024-12-26T04:41:33Z")

</div>

I am having trouble doing hybrid search. to give you guys bit of background I am using semantic search created inference with openai embeddings.

```auto
const addIndex = async () => {
  // try {
  // await client.indices.delete({ index });
  // } catch (err) {
  // //
  // }
  await client.indices.create({
    index,
    mappings: {
      properties: {
        phrase: {
          type: "semantic_text",
          inference_id: "openai_embeddings",
        },
        copilot: {
          type: "keyword",
        },
        org: {
          type: "keyword",
        },
        topic: {
          type: "keyword",
        },
        mode: {
          type: "keyword",
        },
      },
    },
  });
  console.log("created successfully");
};

```

```auto
const response = await client.transport.request({
    method: "PUT",
    path: "/_inference/text_embedding/openai_embeddings",
    body: {
      service: "openai",
      service_settings: {
        api_key: "xxxxxxxx-xxxxxxxx",
        model_id: "text-embedding-3-small",
      },
    },
  });

```

this is my search query

```auto
const response = await client.search({
    index,
    min_score: 0.8,
    query: {
      bool: {
        must: {
          semantic: {
            field: "phrase",
            query: "can i talk to a real person",
          },
        },
        filter: [
          { term: { copilot: "67108e2e8c0611df3ad71109" } }, // Filter by copilotId
          { term: { mode: "DRAFT" } }, // Filter by mode
        ],
      },
    },
  });

```

above query randomly works sometimes and doesnt work other times.  
i make page to 1000 then it started giving results. why is pagesize altering the behaviour? Is it performing post filter? so that means it doesnt work if i have 2000 phrases? Please help me here.

---

<div class="post-metadata">

**Author:** ![ashishtiwari1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashishtiwari1993/32/135241_2.png) [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)\
**Post date:** [December 26, 2024, 8:01am UTC](https://discuss.elastic.co/t/unable-to-perform-hybrid-search/372393/2 "2024-12-26T08:01:00Z")

</div>

Hi @harish32, Welcome to the Elastic community.

The page should not impact to the response.

1. You defining 1000 in size?
2. Any error you getting when there is no response?
3. I am also assuming Async is well handled in your script.

---

<div class="post-metadata">

**Author:** ![harish32](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harish32/32/137750_2.png) [@harish32](https://discuss.elastic.co/u/harish32)\
**Post date:** [December 26, 2024, 10:35am UTC](https://discuss.elastic.co/t/unable-to-perform-hybrid-search/372393/3 "2024-12-26T10:35:27Z")

</div>

no its returning empty array  
here is the query i have ran

```auto
  const response = await client.search({
    index,
    query: {
      bool: {
        must: [
          {
            semantic: {
              field: "phrase",
              query: "talk to a agent",
            },
          },
        ],
        filter: [
          { term: { copilot: "67108e2e8c0611df3ad71109" } }, // Filter by copilotId
          { term: { mode: "DRAFT" } }, // Filter by mode
          { term: { topic: "675a88feb73a5d37e21fc352" } },
        ],
      },
    },
  });
  console.log(response.hits);

```

i have attached the screenshot where i ran same function 3 times you can see it gave result not completely but returned only one then  
3rd time it didnt give anything

 ![Screenshot from 2024-12-26 16-22-16](https://us1.discourse-cdn.com/elastic/original/3X/f/b/fb7c898e2cc5792d3c77e2b2b62b591ccf178187.png)

---

<div class="post-metadata">

**Author:** ![xeraa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xeraa/32/48181_2.png) [@xeraa](https://discuss.elastic.co/u/xeraa)\
**Post date:** [December 27, 2024, 4:36pm UTC](https://discuss.elastic.co/t/unable-to-perform-hybrid-search/372393/4 "2024-12-27T16:36:36Z")

</div>

I would

1. check the Elasticsearch logs (especially for any errors around inference or search)
2. log the actual query run to compare it to the successful ones.

If I had to guess, I'd say turning the semantic query into its dense vector representation fails, so you're not running the same Elasticsearch query in the background. Maybe something like a rate limit is kicking in (but that's just a random guess). I'd be surprised if you ran the exact same Elasticsearch query and it would give you different results (assuming the cluster is healthy).

---

<div class="post-metadata">

**Author:** ![Kathleen\_DeRusso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kathleen_derusso/32/132039_2.png) [@Kathleen\_DeRusso](https://discuss.elastic.co/u/Kathleen_DeRusso)\
**Post date:** [January 2, 2025, 4:47pm UTC](https://discuss.elastic.co/t/unable-to-perform-hybrid-search/372393/5 "2025-01-02T16:47:12Z")

</div>

Another helpful troubleshooting tool could be to run the `knn` search directly on the `semantic_text` field a a nested query. See [example](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-semantic-query.html#advanced-search).

---

<div class="post-metadata">

**Author:** ![harish32](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harish32/32/137750_2.png) [@harish32](https://discuss.elastic.co/u/harish32)\
**Post date:** [January 7, 2025, 10:00am UTC](https://discuss.elastic.co/t/unable-to-perform-hybrid-search/372393/6 "2025-01-07T10:00:03Z")

</div>

I tried knn as well still not getting results

```auto
const response = await client.search({
    index,
    query: {
      bool: {
        must: [
          {
            nested: {
              path: "phrase.inference.chunks",
              query: {
                knn: {
                  field: "phrase.inference.chunks.embeddings",
                  query_vector_builder: {
                    text_embedding: {
                      model_id: "openai_embeddings",
                      model_text: "talk to an agent",
                    },
                  },
                },
              },
            },
          },
        ],
        filter: [
          { term: { copilot: "67108e2e8c0611df3ad71109" } }, // Filter by copilotId
          { term: { mode: "DRAFT" } }, // Filter by mode
          { term: { topic: "675a88feb73a5d37e21fc352" } },
        ],
      },
    },

```

If i remove filter it works fine but i need both to work

---

<div class="post-metadata">

**Author:** ![harish32](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harish32/32/137750_2.png) [@harish32](https://discuss.elastic.co/u/harish32)\
**Post date:** [January 7, 2025, 10:34am UTC](https://discuss.elastic.co/t/unable-to-perform-hybrid-search/372393/7 "2025-01-07T10:34:29Z")

</div>

Thanks for the reply i am new to Elasticsearch. i tried checking logs i cant find anything.

 ![Screenshot from 2025-01-07 15-58-39](https://us1.discourse-cdn.com/elastic/original/3X/0/5/052bbd358dda8e75a3a6f253da41f9abc1847221.png)  
what i believe is happening it is doing a filter first and then searching on the 10 results it filtered. That is why when i increase the page size it is going to return 100 which will contain the semantic word so it is returning matching 10.

I have followed this

> **[Tutorial: semantic search with semantic\_text | Elasticsearch Guide \[8.17\] |...](https://www.elastic.co/guide/en/elasticsearch/reference/current/semantic-search-semantic-text.html)**

Do you have any reference for hybrid search like mine which would be very helpful.

---

<div class="post-metadata">

**Author:** ![Kathleen\_DeRusso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kathleen_derusso/32/132039_2.png) [@Kathleen\_DeRusso](https://discuss.elastic.co/u/Kathleen_DeRusso)\
**Post date:** [January 7, 2025, 1:24pm UTC](https://discuss.elastic.co/t/unable-to-perform-hybrid-search/372393/8 "2025-01-07T13:24:01Z")

</div>

Can you try adding the filters in the knn filter section, to pre filter the results returned by knn instead of post filtering?

---

<div class="post-metadata">

**Author:** ![harish32](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harish32/32/137750_2.png) [@harish32](https://discuss.elastic.co/u/harish32)\
**Post date:** [January 8, 2025, 4:48am UTC](https://discuss.elastic.co/t/unable-to-perform-hybrid-search/372393/9 "2025-01-08T04:48:36Z")

</div>

Thank you so much @Kathleen_DeRusso 🙏. It is working now.  
Thanks @xeraa for jumping in here

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 5, 2025, 4:49am UTC](https://discuss.elastic.co/t/unable-to-perform-hybrid-search/372393/10 "2025-02-05T04:49:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.