Vector search in Enterprise Search over documents from Confluence Server

Hi all,

I am evaluating vector search functionality over Confluence documents in Elastic Enterprise Search. I'm using the built-in Confluence Server Connector to ingest all documents from Confluence, and also plan to use the search UI built into Workplace Search for searching the documents based on the vector embeddings. (At least my idea is to use as many of the built-in components as possible for the initial experiments to test if ML brings better search accuracy for my documents.)

In the "Add inference processors to your ML inference pipeline" chapter of the "Document enrichment with ML" I found a guide on how to set up an ingestion pipeline with ML to enrich the ingested contents using Enterprise Search.

According to this document I should be able to see all my indices and contents and set up the ML Pipeline in the Enterprise Search → Content → Indices menu in Kibana, however this page does not list any contents even after I ingested all our documents from confluence. I'm wondering if this might be related to the fact that the connector uses streams and not indices directly, but I could not figure out why this page is empty.

Regarding this topic, I'd have two questions:

  • Can the built-in search UI of Workplace Search be used to search the vectors?
  • If so, do I have to use the tools provided by Workplace Search to set up the ML pipeline (e.g. at Enterprise Search → Content → Indices), or can it be tailored to work with custom vector fields? I think as a workaround I could create an ingest pipeline on my own and set it as default but I fear with this I'd lose the ability to use the built-in UI provided by Workplace Search.

Thank you and regards,
Dávid

1 Like

i was not able to find any thing related as well

Hi Dodi and Abdurrahman, sorry for making you wait for a reply.

The Confluence Server Connector you're using is in Workplace Search, which does not use the ML inference pipeline. Instead, you'll want to use our newer connector clients to make this work smoothly.

2 Likes

thanks for your reply,
so is it possible to connect ServisNow and include a vector field representing embedding.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.