Search across index

Elastic Stack Elasticsearch
1

Dear Eleasticsearch team,

My team is facing an interesting situation. There are two web services - "Account" and "Membership". "Account" service stores customers' names, gender, date_of_birth, etc. "Membership" service stores which product plan customers have subscribed. Both of them plan to use Elasticsearch to provide searching function.

For some reasons, we could not merge these two services (or their data models) together. We can only design how to create index, define index mapping, and build search query.

One idea is each service has its own index. Then each index's mapping can perfectly match each service's data model. And searching query will be simple and straightforward.

But there is only one searching UI. Again my team cannot change it either. For example, admins can search on "first_name = Tom AND plan = Apple Mobile". It will generate two different searching queries on two indices. We can return the inner set of two searching results. But if we want to provide features like pagination, it will be quite difficult.

Could we apply different queries on different indices, but Elasticsearch returns one result? The "customer_id" is the common field of documents in two indices. For example,

(POST) /account,membership/doc/_search?size=50

{
  "query": [
    {
      "index": "account",
      "query_string": {
        "query": "first_name: will AND gender: male"
      }
    },
    {
      "index": "membership",
      "query_string": {
        "query": "plan: Apple Mobile"
      }
    }
  ]
}

I know the above example does not match your syntax. I just use it to describe my question.

Another solution is merging two indices into one indices. Then we could simply rely on Elasticsearch built-in searching feature and scoring mechanism.

But we still would like to try a bit more with two indices, separate search and merge/aggregate two searching results within Elasticsearch.

Sorry for a long message.

Regards!

2

Not really. The closest feature you have is Multi search API | Elasticsearch Guide [8.11] | Elastic but it's like running:

GET /account/_search?size=50
{
  "query": 
    {
      "query_string": {
        "query": "first_name: will AND gender: male"
      }
    }
}
GET /membership/_search?size=50
{
  "query": 
    {
      "query_string": {
        "query": "plan: Apple Mobile"
      }
    }
}

So results won't appear as a single document.

You might want to look at parent/child feature, although I'm not a big fan of it as it comes with some drawbacks, like memory pressure, more complex queries, slower queries...

As you suppose, it's definitely better to send in elasticsearch data in a way that makes the search easier, faster, better... :slight_smile:

3

Thanks, Dadoonet.

I would like to avoid applying parent/child feature.

So I guess you may also recommend us to create one index for two different web services. The index mapping is a combination of two services' data models. Then we could create one simple and straightforward query to search.

4

I'd totally forget about the existing model and would just think about the usage to build the right "search" objects for my use case.

Basically I'd recommend to ask yourself 2 questions:

  • What kind of objects my users want to get back as a response? If it's object X, then just index object X
  • What typical attributes my users want to search for? Let say I need attribute a, b and c, just index those attributes within object X whatever the original source of those attributes is.

HTH

5

Hi, Dadoonet.

I think what I try to ask is how to just index the object my users want.

For example, my users want to receive an object made of "name", "gender" and "plan_id". Service A maintains "name" and "gender". But service B maintains "plan_id".

Do I create one index and allow two services to create/update/delete documents into this shared index?

Any special configuration or feature that I should set up for this scenario?

Thanks!

6

Are you sure that your end user would like to get an id ?
I'd think he would prefer getting back a service, not the id of the service.

But a more concrete example would help.

7

Oh, Dadoonet, you are right.

This is my current idea:

  1. "Account" service creates a new account and index it into Elasticsearch
  2. "Membership" service assigns a plan to an account, and update Elasticsearch
  3. "Search" service searches on Elasticsearch

Web Request examples (in order to make my message short, I may not fully follow the syntax)

  1. "Account" service creates a new account and index it into Elasticsearch
(POST) /index/doc/user_123

{
    "id": "user_123",
    "name": "will",
    "gender": "male"
}

(POST) /index/doc/user_456

{
    "id": "user_456",
    "name": "william",
    "gender": "male"
}
  1. "Membership" service assigns a plan to an account, and update Elasticsearch
(POST) /index/doc/user_123/_update

{
    "plan": "bupa_123"
}

(POST) /index/doc/user_456/_update

{
    "plan": "bupa_456"
}
  1. "Search" service searches on Elasticsearch
(GET) /index/doc/_search

{
    "query": {
        "query_string": {
        	"query": "name: (will) AND plan: (bupa_123, bupa_456)"
        }
    }
}

Expected result shall be
{
  "results": [
    {
      "id": "user_123",
      "name": "will",
      "gender": "male",
      "plan": "bupa_123"
    },
    {
      "id": "user_456",
      "name": "william",
      "gender": "male",
      "plan": "bupa_456"
    }
  ]
}
8

According to the desired output, I think that the object you want to search for is a plan.

So I'd insert a plan exactly as this:

PUT plans/_doc/bupa_456
{
  "id": "user_456",
  "name": "william",
  "gender": "male",
  "plan": "bupa_456"
}
PUT plans/_doc/bupa_123
{
  "id": "user_123",
  "name": "will",
  "gender": "male",
  "plan": "bupa_123"
}

See what I mean?

9

Hi, Dadoonet.

In my previous example, "plan" is one detail of users. There are more searchable user details. That is why I use "account" as document, and we have two different web services to maintain them.

More information about our data models:

  • "Account" service stores "first_name", "last_name", "gender", "date_of_birth" and "email". All fields are mandatory. They cannot be null or empty.

  • "Membership" services stores "plan_id", "consumer_network_id", "preferred_branch_id", "customer_manager_id" and "status". All fields are optional. They can be null or empty.

  • "Search" service wants to search on any of the above 10 attributes.

Therefore, at beginning, I think about two web services maintains their own index separately (create / update / delete documents). Then a "Search" service searches on two indices and merges searching result.

Later I realise the above idea has natural difficulty for pagination and require extra work on merging. Then I switch to two services maintain the same index, and start to ask for suggestions here.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.