Hi all,
I am evaluating vector search functionality over Confluence documents in Elastic Enterprise Search. I'm using the built-in Confluence Server Connector to ingest all documents from Confluence, and also plan to use the search UI built into Workplace Search for searching the documents based on the vector embeddings. (At least my idea is to use as many of the built-in components as possible for the initial experiments to test if ML brings better search accuracy for my documents.)
In the "Add inference processors to your ML inference pipeline" chapter of the "Document enrichment with ML" I found a guide on how to set up an ingestion pipeline with ML to enrich the ingested contents using Enterprise Search.
According to this document I should be able to see all my indices and contents and set up the ML Pipeline in the Enterprise Search → Content → Indices menu in Kibana, however this page does not list any contents even after I ingested all our documents from confluence. I'm wondering if this might be related to the fact that the connector uses streams and not indices directly, but I could not figure out why this page is empty.
Regarding this topic, I'd have two questions:
- Can the built-in search UI of Workplace Search be used to search the vectors?
- If so, do I have to use the tools provided by Workplace Search to set up the ML pipeline (e.g. at Enterprise Search → Content → Indices), or can it be tailored to work with custom vector fields? I think as a workaround I could create an ingest pipeline on my own and set it as default but I fear with this I'd lose the ability to use the built-in UI provided by Workplace Search.
Thank you and regards,
Dávid