# Latest unique transform missing documents

**URL:** <https://discuss.elastic.co/t/latest-unique-transform-missing-documents/368700>\
**Category:** Elastic Search\
**Tags:** transforms\
**Created:** [October 11, 2024, 7:02pm UTC](https://discuss.elastic.co/t/latest-unique-transform-missing-documents/368700 "2024-10-11T19:02:56Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Phil\_McLachlan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phil_mclachlan/32/124424_2.png) [@Phil\_McLachlan](https://discuss.elastic.co/u/Phil_McLachlan)\
**Post date:** [October 11, 2024, 7:02pm UTC](https://discuss.elastic.co/t/latest-unique-transform-missing-documents/368700/1 "2024-10-11T19:02:57Z")

</div>

We have what I believe to be a straightforward transform to pick out unique documents ordered by an ingest pipeline timestamp. We apply it to several programs (customers) index data, but with larger data it is missing documents. With 5000 or less documents in an index it seems to work fine, but with 23000 documents it seems to be missing about 500 to 1000 documents.

Furthermore, I have noticed that when the timestamps are close, it is not picking the latest one. It picks the earliest one.

Here is the transform:

```auto
PUT _transform/phils_test_unique
{
  "source": {
    "index": "axp_marketplace_search_catalog_1_1728534453"
  },
  "dest": {
    "index": "phils_test_unique",
		"pipeline": "axp_marketplace_event_ingested_ingest_pipeline"
  },
	"latest": {
		"unique_key": ["product_pk", "catalog_type"],
		"sort": "event.ingested"
	},
	"description": "",
	"frequency": "5m",
	"sync": {
		"time": {
			"field": "event.ingested",
			"delay": "60s"
		}
	}
}

```

Can anyone tell me why the resulting index will be missing documents? It seems to be missing the same amount each time it is run.

---

<div class="post-metadata">

**Author:** ![Phil\_McLachlan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phil_mclachlan/32/124424_2.png) [@Phil\_McLachlan](https://discuss.elastic.co/u/Phil_McLachlan)\
**Post date:** [October 11, 2024, 9:18pm UTC](https://discuss.elastic.co/t/latest-unique-transform-missing-documents/368700/2 "2024-10-11T21:18:05Z")

</div>

Ok, I figured it out. I had to make the input index definition of catalog\_type to be of type "keyword", and then I had to remove the keyword suffix from the unique\_key in the transform. Now all the documents are in the output index.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 8, 2024, 9:18pm UTC](https://discuss.elastic.co/t/latest-unique-transform-missing-documents/368700/3 "2024-11-08T21:18:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.