# Deduplication - Collapse - Pagination

**URL:** <https://discuss.elastic.co/t/deduplication-collapse-pagination/381928>\
**Category:** Elastic Search\
**Created:** [September 14, 2025, 6:01am UTC](https://discuss.elastic.co/t/deduplication-collapse-pagination/381928 "2025-09-14T06:01:59Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![gurcharan\_singh](https://avatars.discourse-cdn.com/v4/letter/g/dec6dc/32.png) [@gurcharan\_singh](https://discuss.elastic.co/u/gurcharan_singh)\
**Post date:** [September 14, 2025, 6:01am UTC](https://discuss.elastic.co/t/deduplication-collapse-pagination/381928/1 "2025-09-14T06:01:59Z")

</div>

Hi,

I have two data sources - Global (14M records) & customer specific (5M records)  
Data - Products with fields - name, description, manufacturer part number (MPN), vendor sku

Some search query.  
Output -  
Results grouped over MPN with relevancy (most matching doc score decide relevancy of group) from all data sources with deduplication across pages and pagination over grouped results.

Can I use Collapse on MPN with my search query on Unified Data model (differentiating docs by source\_type ) here ? with nearly 22M records for fast retrieval.

Is there any other way to do it ?  
I can’t pre group global data with customer specific data on MPN need that separation as there can be n customers and single doc can be very large.

Is there any other way to avoid grouping over Global Data source docs as its huge data. Some other ways with queries?

Unified data models -

Product from Global data source (no company\_id)

```auto
{"mpn": "DELL-G7GV0","company_id": null,"source_type": "icecat","source_id": "80076143","product_name": "DELL G7GV0","description": "DELL G7GV0. Brand compatibility: Dell","manufacturer": "DELL","vendor_sku":"244545"}

```

Customer specific data

```auto
{"mpn": "DELL-G7GV0","company_id": "f1eb99a720ef4f05bfa6beae16aa235c","source_type": "custom_distributor","source_id": "34565434","product_name": "DELL G7GV0","description": "DELL G7GV0. Brand compatibility: Dell","manufacturer": "DELL","vendor_sku":"2345654"}  

```

@stephenb

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [September 15, 2025, 3:37pm UTC](https://discuss.elastic.co/t/deduplication-collapse-pagination/381928/2 "2025-09-15T15:37:16Z")

</div>

Hi @gurcharan_singh Welcome to the community.

A couple house keeping items

First please do not `@` people in your topic that have not already joined, doing this is consider not best forum etiquette. And doing so can turn off / limit the people that may answer the question... and in fact this is not my area of expertise 🙂 so you have already limited your chance of success....

What Version are you running?

You are asking a really broad question like

> Can I use Collapse on MPN with my search query on Unified Data model (differentiating docs by source\_type ) here ? with nearly 22M records for fast retrieval.

Fast Retrieval can mean many things  
You are speaking of your data like we all understand it 🙂

Also you will need to provide further examples like a couple documents and what your expected results.

That will usually help ... the more you provide the more likely you will get help.

I may or may not be able to help but perhaps with more info someone else can