Hi All,
I was wondering if anyone knows if there is a way to get a custom index as an Organizational Source within Workplace Search?
Background, I have an Enterprise Search web crawler setup to index a website into an index called search-knowledgebase. I'd like to make this index and its contents searchable within the Workplace search application however I can't really find a way to add a "custom index" as a source to workplace search. Would anyone know how to do this?
Hi @BenB196,
That's not directly possible today. You could us the Workplace Search Custom Source API to attempt to mirror the data from your Crawler index into a Workplace Search Content Source, but that would result in you storing the data twice.
Are you using Workplace Search for other content sources? Is that why you want to have Crawler data in Workplace Search, is because you want to search multiple data sources in one place?
If that's the case, you may be better served going in the other direction, and switching to use Elastic Connectors, which are the next evolution of Workplace Search. In fact, in recent versions, the product really tries to guide you away from using Workplace Search, as the connectors are a lot more transparent, have stronger performance metrics, a larger catolog, and more active support/development.
If you had data from Connectors and Crawler, you could search over all of it using either an App Search Meta Engine or a Search Application with multiple backing indices.
If neither of those are a good fit, can you explain more about your use case?
Hi @Sean_Story thanks for the reply. We're currently exploring the Workspace Search app within Kibana and we plan on using it with a number of products (Jira/Sharepoint/etc.). One of the challenges is that we have a number of more traditional websites that we want to allow people to search against and that is why we're testing out the webcrawler.
Regarding, the "new" approach of Elastic Connectors. This is where I'm honestly confused about the Elastic product stack. These connectors make sense functionally, but it isn't clear on how/if they can integrate with the Elastic Workplace Search UI.
The documents I've found seem to indicate the Elastic Connectors require developing custom UIs. This isn't something we're really interested in doing. We were more looking to just use the Out-of-the-box Workplace search UI and just add additional data sources.
Is it possible to use the new Elastic Connectors with the Workplace search UI, or is it correct in saying we'd need to develop our own UI for those?
This is where I'm honestly confused about the Elastic product stack.
Totally understandable. We've got a small-ish team, and pivoting the product, marketing, and docs in a way that still supports the legacy systems but tries to ween customers off of them is hard. I appreciate your patience and perseverance. Also very interested if you can provide any concrete details on things you've seen/read/experienced that added to the confusion, and/or insights on what could help improve the experience.
These connectors make sense functionally, but it isn't clear on how/if they can integrate with the Elastic Workplace Search UI.
They cannot. The Workplace Search OOTB search experience is tightly coupled to Enterprise Search (the server) and Workplace Search (the product). It's also not super configurable, and a common ask has been to re-style it or embed the search experience elswhere, so that your search users don't know they're using "Elastic Workplace Search" but just "Acme Inc's internal search bar". So we found that most people weren't served well by the OOTB UI.
Your use case of having crawler data that you want to search alongside jir/sharepoint data is one of the key motivators for why we started building Connectors that are closer to the Elastic stack primitives (direct access to index settings, mappings, auth, etc). Workplace Search came to Elastic through an acquisition, and has been in a long journey of assimilating closer and closer to stack primitives.
The documents I've found seem to indicate the Elastic Connectors require developing custom UIs.
This is true. However, with tools like Search-UI and Build a search experience with the Search Application client | Elasticsearch Guide [8.11] | Elastic, a lot of this gets bootstrapped for you, and it's pretty quick to build a search experience. This also makes it much easier for customers to style their search experience and host/embed it wherever they want, as per my earlier points on difficulties we had with the Workplace Search UI.
We were more looking to just use the Out-of-the-box Workplace search UI and just add additional data sources
Have you looked at App Search at all? Because it gives you an easy button that generates a whole web-app (powered by App Search and Search UI). It has a management UI in Kibana (a lot like Workplace Search) but decouples the end-user search experience from Kibana/Enterprise search. I suggest you check this out, as it may meet your needs without locking you into a product that's probably headed towards deprecation.
I'm an Elastic employee, but will also certify this approach as a customer - I am not a frontend developer and find myself very intimidated by React. But I was able to build and deploy a pretty functional search experience for a personal project/hobby using this approach.
Hi @Sean_Story
Also very interested if you can provide any concrete details on things you've seen/read/experienced that added to the confusion, and/or insights on what could help improve the experience.
I think my main "issue" here is how close together all of these products are within the docs, with no explicit mention of what works together with what. If you look at the side bar of What is Elastic Enterprise Search? | Enterprise Search documentation [8.11] | Elastic, you'll see "Connectors" and "App search and Workplace Search" both there, but nothing that explicitly states on the "Connector" pages that they don't function with Workplace Search.
We've got a small-ish team, and pivoting the product, marketing, and docs in a way that still supports the legacy systems but tries to ween customers off of them is hard.
Definitely understand this.
It's also not super configurable, and a common ask has been to re-style it or embed the search experience elswhere, so that your search users don't know they're using "Elastic Workplace Search" but just "Acme Inc's internal search bar". So we found that most people weren't served well by the OOTB UI.
This definitely makes sense, and in our case long-term this would be valuable, but one of the issues we're looking to address is "time to value". Being able to leverage the OOTB Workplace search as a jumping off point to show the value, then working on a bespoke app, is a far easier selling point, then saying we need to develop something from the start go through security implementation and auditing and deploy a completely fresh product, since Elastic has already done most of this with the OOTB product.
That's not directly possible today. You could us the Workplace Search Custom Source API to attempt to mirror the data from your Crawler index into a Workplace Search Content Source, but that would result in you storing the data twice.
I don't suppose you'd have any link on how to do something like this? From our end, we're not talking about a lot of data here, so duplicating it isn't too bad of a concept if it is an "easy-win".
Or maybe another way to look at this, does Elastic have an example application which already implements SAML and DLS for the new connectors? That would potentially be another way to "easy-win" here, as that is primarily the functionality we're leveraging via the OOTB solution, SAML + DLS.
you'll see "Connectors" and "App search and Workplace Search" both there, but nothing that explicitly states on the "Connector" pages that they don't function with Workplace Search.
This is a great insight, thank you. I'll raise this with our Docs and Product Managment teams, and we'll see if we can make the interrelations (or lack thereof) more clear across our offerings.
one of the issues we're looking to address is "time to value".
Totally makes sense. I'd encourage you to play for ~30 min with App Search and Search UI before you commit to the Workplace Search UI. I think it might solve the time-to-value need you have. If not, I can respect how Workplace Search is more attractive right now - just be prepared that you won't have an easy migration path off of Workplace Search once you are committed to it, since it buries its relationships to Elastics primitives (like the Elasticsearch Index and _doc APIs) deep under the hood.
You could us the Workplace Search Custom Source API to attempt to mirror the data from your Crawler index into a Workplace Search Content Source
don't suppose you'd have any link on how to do something like this?
Lots of ways you could accomplish this. If you wanted to use all Elastic ecosystem components, you could use Watcher with a Search Input and a Webhook Action . This could, on a cron-like schedule, see what data is in your Elastic Connector index and then use the webhook to POST that data to the Workplace Search Custom Source API. And right after, you could have another webhook trigger to Delete by Query any docs in your content source that weren't just updated.
I don't have a blog or anything to link you to though, because again, we're actively trying to encourage folks away from Workplace Search, not towards it.
does Elastic have an example application which already implements SAML and DLS for the new connectors?
Yes and no? Elasticsearch supports SAML as an authentication realm. See: Configuring SAML single-sign-on on the Elastic Stack | Elasticsearch Guide [8.11] | Elastic
and a number of our Connectors (including Jira and Sharepoint Online) support DLS. See: Document level security | Enterprise Search documentation [8.11] | Elastic
What's not fully baked at the moment is the interplay between the two. You'd need to craft an Elasticsearch Role Mapping based off of SAML metadata, or build custom logic in your own backend to generate user-specific API Keys/Roles, in order to make sure that they can't search something they shouldn't.
This may sound complex, but keep in mind that the Workplace Search analog is the External Identities API, which requires manual upkeep and 1 request-per-user to map their search user identity to the 3rd-party (jira/sharepoint) identities that DLS is based off of. This has historically been another significant pain point for production use cases of Workplace Search with DLS.
Thanks for the insight here @Sean_Story. I'll take a deeper look at the App Search and Search UI functionality to try and get a better understanding here of what it will really take to get to a usable product.
Hi @Sean_Story one question, you mention App Search, and by that I assume you mean; this, but there is also Search Applications. This guide, Leverage document-level security from connectors in Search Applications | Enterprise Search documentation [8.11] | Elastic, mentions the later, is it "more correct" to use Search Applications here rather than App Search for this type of solution?
is it "more correct" to use Search Applications here rather than App Search for this type of solution?
you just want to open all the cans of worms, don't you? 
App Search, like Workplace Search, is a product that is hosted in the Enterprise Search Server. These products came from an acquisition of Swiftype back in the day. They're both good products, but they were built as consumers of Elastic products, not as native products. So there's always been this tension where power users struggle to leverage the Elasticsearch primatives when they're using Enterprise Search products.
The migration path away from Workplace Search is clear, and while it doesn't have a concrete deprecation timeline yet, it's pretty obvious that it's coming.
App Search has historically been much more popular, and therefore is going to take a lot more from us to provide a migration path to all native Elasticsearch stack primitives. Search Applications (we're running out of names, aren't we?) are a first step towards eventually offering all of App Search's great features as an Elasticsearch-centric product. Some things (synonyms, curations, analytic dashboards, meta-search) have been replicated. Other things (UI generating easy-button, modern relevance OOTB, opinionated mapping defaults) have not been.
Search Applications are definitely a valid product for you to investigate. They'll probably set you up for longer-term (decades vs years) success. But you also might have to do some heavier lifting. And choosing App Search now won't be as problematic as choosing Workplace Search now, because it's large popularity will ensure a really smooth migration path if/when it does come to an end-of-life (for which there is none on any horizon I'm aware of).
If you look into Search Applications, instead of Search UI, you'll want to look at the Search Application Client, which is the react libarary analog for bootstrapping a modern search experience.
Since I've decided we're now friends, I'll share a link to this (admittedly still rough, very much just an example) example app that uses Flask+Search Application Client (react) to show off how to search multiple connectors, similar to how Workplace Search provides a UI for multiple content sources. I'll eventually write a blog that goes along with this app. Stay tuned to the Elastic Search Labs Blog.
DO NOT just copy this code. This is for inspiration only, and comes without warranty. But I figure it might help you bridge the gap between the Search Application Client docs and reproducing the Workplace Search UI.
Thanks for the detailed explanation on this @Sean_Story on the two products (as well as the additional example).
I have a two follow up questions after looking into Search Applications a bit more.
beta, do you expect this area to be graduating the GA in the near future, or is it at least somewhat stable?what is the "maturity" of the Search Application feature?
I can't offer a timeline for when it will be GA, but my impression is that it's pretty stable. TBH I didn't even know it was still in beta. But Elastic's definition of "beta" is that it may be subject to change but isn't going to be removed - so there definitely is a future for it.
The process flow around API keys seems a bit cumbersome
I think you mean specifically around Connector DLS? Yes, that's cumbersome today. We have future plans to make it easier to use Authentication metadata to map a user directly to a role that includes the right filters for their identity. You may notice that Connector DLS is also currently in beta. I can't give you a timeline on this either, but can assure you it's very top-of-mind to my team.
Thanks again @Sean_Story!
Regarding this one, and my question was worder pretty badly, but I guess, given that the search application API is native to Elasticsearch, any of the user authentication methods will work with the API, and not just the API key method?
given that the search application API is native to Elasticsearch, any of the user authentication methods will work with the API, and not just the API key method?
Yep! API keys can be created with a Role Descriptor, which is effectively what the connector Access Control syncs produce. But you could use any elasticsearch authentication mechanism, and then set up Role Mappings and create concrete Roles from those Role Descriptors too. The important bit is just making sure that the request automatically gets associated with the specific filter that the connectors produced in the Access Control Sync.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.