Hi David and Mark,
First of all, a big sorry for this very late reply. Meanwhile, I was doing other stuff, not related to this (interesting) topic, and that kept me from answering (because I also wanted to provide you a decent reply )
As a quick reply; yes, this certainly help. Thanks already for that.
But, Mark (and David), like you ask, I'll supply a bit more context so that you guys have an better idea of the question or challenge here.
First a bit of state and/or context: we have a specific product (used by a few thousand dedicated users). And functionality is growing as we speak. Classic web-front combined with several REST backend services. Content is persisted in Relational databases. So, classic relations and we talking about like 20 million of records here and growing.
Besides that content basis (rel dbs), we also want to make this content (or a significant subset of it) high searchable. Enter Elastic (over a year ago), so we're already acquainted with indices, mappings, indexing, querying, shards, scripting, etc. And we've already indexed (these) 20 mio as json docs to Elastic.
Besides that, we have a Search component that fires (api) requests (the usual suspects: _search, aggregations, full-text search etc). Works fine.
BUT and this is the challenge here, of course. Those indexed docs (let's call this index 'job' ) contain values/terms, that are like 'foreign keys' to docs in the same or in other indices in the cluster. And, besides that, we would also like to index the organisational structure of the compagny; think of individuals in their teams and teams-in-teams in an Organisational structure (let's call this index 'organisation' ). Mostly arranged as a tree-structure, but not necessarily: we can also have cross-team compositions, like 'workgroups' and such. More-over, certain terms in the 'job' index refer foreign-key like to unique entries in the 'organisation' index.
A 'job' can have many properties on itself, as has the 'origanisational entry'. So, this content also needs to be highly searchable and displayed eventually.
So, to make it more tangible: a job can have a (sub-)job relation, and each jobs is executed by more 'individuals and/or teams'. It's interesting to know which jobs are assigned to whom and/or find jobs that are related to a certain team (anywhere in the 'organisation').
Besides that, and this is a valuable use-case imo, it would also be nice to graphically show the organisation and be able to travers it, eventually ending up in more detail of the person or team and details on their related jobs.
And that's where, imho, a graph approach and the typically traversal possibilities come in. In other words: be able to meander and drill-down or -up during that journey.
This feels like a job for a Knowledge graph product, but we also need the capabilities to do full and fast searching. Hence why Elastic (search and potentially Graph (explore api and UI) ) and combi with Neo4j came into my view of interest
It could be something like a combi: store detailed data in Elastic indices, store relations (triplets) from that data into neo4j. And use this approach to handle the use-cases described above.
There might be other interesting roads or combinations to discover here (maybe also using different products or integrations). But these especially came into mind.
Voila, the full story. Hope this sheds some light on the challenge (and nice opportunity here ).
So, any advice or thoughts from you guys are very welcome !!
PS: we've already experimented with 'joins' in Elastic (and than will not be the solution) + I already have a fair knowlegde of Neo4j (and Cypher).