Elastic Mapping For Search and Updates

Elastic Stack Elasticsearch
1

Hi,

I am building an Enterprise level search for a Digital Asset Management where I can have below Entities

Asset

id, name, source, owner, expirationdate , tags

Each of these assets can be tagged to i items , campaigns and offers each of which have their own attributes

item - id, description, active , status

campaign - id, name, startDate, endDate, status

offer - id, name, type, startDate, endDate, status

From an Elasticsearch persecpective, Need to find ways to store it effectively. If we store as one big document, it can grow and will cause performance issues. Updates will be large as well

If we store as mapping , containing a single item document containing item info plus assetId, update is smooth but search needs two hops

Is their some way to store it and keeping search faster with low operational overhead ?

2

Hi Harinder_Singh, welcome back. Good to see you here again.

For this type of DAM search use case, I would not try to model it like a relational database inside Elasticsearch.

If users are mainly searching for assets, I would usually keep one search document per asset and denormalize only the fields needed for search, filtering, sorting and display. The full item, campaign and offer data can still live in the source system, or in separate indices if you need to manage/search them independently.

The trade-off is that when an item, campaign or offer changes, you need to update the affected asset documents. But in practice, that is often simpler and faster than relying on query-time joins.

I would only use nested if you really need to preserve the relationship between fields inside each related object. I would avoid parent/child unless there is a very specific reason, because the join field adds query-time overhead. Elastic also calls this out in the documentation:

So my starting point would be: one optimized assets_search index, one document per asset, with only the related fields that are actually needed for the search experience.

3

Hi @Rafa_Silva

Thanks for the prompt suggestion.

for keeping every info in a single projection index, an asset can have close to 100k item associations alone which will likely increase the index size as search can happen on any item value

4

That detail is important.

With ~100k item associations per asset, I would not keep all item values inside the asset document. The document would become too large, and every related update would be expensive.

I would split this into two search shapes:

`assets` for the asset metadata, and another index for the asset-item associations, with one document per association containing `asset_id` plus the searchable item fields.

Then search the association index and return the matching assets from there, either by collapsing on `asset_id` or by fetching the final asset documents in a second step.

Field collapse may help if the final result should still be one result per asset:

Reference

I would only consider `parent/child` after testing this approach, not as the default design, because `has_child` queries add join cost at query time

Reference

5

In case where it needs to match same documents from different different indices, the application will have to do the intersection logic.

Can elastic help here , not from JOIN but a same query able to give matching results from different indices (say group by assetId)

6

Yes. I use a similar ES|QL pattern in my day-to-day work to correlate e-commerce orders and cancellation events stored in separate indices.

For example:

FROM ecommerce-orders, ecommerce-cancellations METADATA _index| WHERE (_index == "ecommerce-orders" AND order_status == "created")OR (_index == "ecommerce-cancellations" AND cancellation_status == "requested")| STATS order_matches = COUNT() WHERE _index == "ecommerce-orders",cancellation_matches = COUNT() WHERE _index == "ecommerce-cancellations"BY order_id| WHERE order_matches > 0 AND cancellation_matches > 0| KEEP order_id

This returns only the order_id values found in both filtered datasets. The same pattern can be applied to your assetId.

It is not a join and does not merge the documents, but it can move the intersection logic from the application into Elasticsearch. Just make sure the shared field has a compatible mapping across the indices, preferably keyword.