# 'Latest' Transform job not refreshing all docs in the destination index

**URL:** <https://discuss.elastic.co/t/latest-transform-job-not-refreshing-all-docs-in-the-destination-index/305720>\
**Category:** Elasticsearch\
**Tags:** transforms\
**Created:** [May 26, 2022, 2:12pm UTC](https://discuss.elastic.co/t/latest-transform-job-not-refreshing-all-docs-in-the-destination-index/305720 "2022-05-26T14:12:26Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![ykara84](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ykara84/32/106391_2.png) [@ykara84](https://discuss.elastic.co/u/ykara84)\
**Post date:** [May 26, 2022, 2:12pm UTC](https://discuss.elastic.co/t/latest-transform-job-not-refreshing-all-docs-in-the-destination-index/305720/1 "2022-05-26T14:12:26Z")

</div>

Hi, I'm new to Transforms.

I have a Powershell script that every hour gathers VMware virtual machine capacity metrics (cpu, memory and such) from vCenter and stores them in an index: virtualisation-vm-_yyyy-MM_

I've created a simple **latest continuous** Transform Job that sends the latest records from the index-pattern _virtualisation-vm-_\* to a destination index called _virtualisation-latest-vm-vsphere_. I thought it was working as expected but after a few days I've noticed **some** VMs in the latest destination index have not updated since the initial transform job was created even though in the original index new data has been added for those VMs every hour since the Transform was created

I've recreated the Transform job several times with different settings (freq, delay, etc) but with no difference. How do I debug this? There's no errors or warnings in the messages section of the Transform

again this does not affect all docs, the majority update themselves, but about 10% don't and it doesn't always seem to be the same ones that don't update when I recreate the transform job

Below are my settings:

(3 node cluster)

```auto
value={
  "id": "virtualisation-latest-vm-vsphere",
  "version": "7.16.2",
  "create_time": 1653387161554,
  "source": {
    "index": [
      "virtualisation-vm-*"
    ],
    "query": {
      "match_all": {}
    }
  },
  "dest": {
    "index": "virtualisation-latest-vm-vsphere"
  },
  "frequency": "60m",
  "sync": {
    "time": {
      "field": "@timestamp",
      "delay": "60s"
    }
  },
  "latest": {
    "unique_key": [
      "vm.keyword"
    ],
    "sort": "@timestamp"
  },
  "description": "Only the latest information from the virtualisation capacity data.",
  "settings": {
    "max_page_search_size": 500
  },
  "retention_policy": {
    "time": {
      "field": "@timestamp",
      "max_age": "30d"
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![sophie\_chang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sophie_chang/32/18008_2.png) [@sophie\_chang](https://discuss.elastic.co/u/sophie_chang)\
**Post date:** [May 26, 2022, 3:11pm UTC](https://discuss.elastic.co/t/latest-transform-job-not-refreshing-all-docs-in-the-destination-index/305720/2 "2022-05-26T15:11:02Z")

</div>

From initial information, the most likely explanation would be that there is a divergence between `@timestamp` and the time of ingest (however I'm not sure how long a `sync.time.delay` you experimented with).

As a best practice, the sync time field should be the time of ingest. This is the best way for transforms to be able to identify changes since the last time it checked. This can be set using an ingest processor, something along the lines of:

```auto
PUT _ingest/pipeline/set_ingest_time
{
  "description": "Adds ingest timestamps",
  "processors": [
    {
      "set": {
        "field": "_source.@timestamp_ingest",
        "value": "{{_ingest.timestamp}}"
      }
    }
  ]
}

```

The best way to see if you are getting divergence between `@timestamp` and ingest time, would be to plot counts for both.

Also there could perhaps be errors relating to writing to the index `virtualisation-latest-vm-vsphere` in the Elasticsearch logs.

Hope this helps  
Sophie

---

<div class="post-metadata">

**Author:** ![ykara84](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ykara84/32/106391_2.png) [@ykara84](https://discuss.elastic.co/u/ykara84)\
**Post date:** [May 27, 2022, 9:02am UTC](https://discuss.elastic.co/t/latest-transform-job-not-refreshing-all-docs-in-the-destination-index/305720/3 "2022-05-27T09:02:27Z")

</div>

Thanks Sophie

Can I have more info on how I can set the Ingest Timestamp for my index-pattern or point me to help/tutorial page? I've never used pipeline processors before

---

<div class="post-metadata">

**Author:** ![ykara84](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ykara84/32/106391_2.png) [@ykara84](https://discuss.elastic.co/u/ykara84)\
**Post date:** [May 27, 2022, 9:50am UTC](https://discuss.elastic.co/t/latest-transform-job-not-refreshing-all-docs-in-the-destination-index/305720/4 "2022-05-27T09:50:59Z")

</div>

I've managed to create the pipeline using Sophies example, but cant see anywhere to tell the pipeline to only apply that ingest timestamp fields to my original index-pattern "virtualisation-vm-\*"

how does it know which indexes to apply the pipeline to? does it just apply it to all indexes?

---

<div class="post-metadata">

**Author:** ![sophie\_chang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sophie_chang/32/18008_2.png) [@sophie\_chang](https://discuss.elastic.co/u/sophie_chang)\
**Post date:** [May 30, 2022, 8:57am UTC](https://discuss.elastic.co/t/latest-transform-job-not-refreshing-all-docs-in-the-destination-index/305720/5 "2022-05-30T08:57:04Z")

</div>

There are a few examples here which explain how to use a pipeline when indexing documents into the `*virtualisation-vm-* *` index - which is the transform source index. [Ingest pipelines | Elasticsearch Guide [8.2] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html#add-pipeline-to-indexing-request)

---

<div class="post-metadata">

**Author:** ![ykara84](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ykara84/32/106391_2.png) [@ykara84](https://discuss.elastic.co/u/ykara84)\
**Post date:** [May 30, 2022, 9:38am UTC](https://discuss.elastic.co/t/latest-transform-job-not-refreshing-all-docs-in-the-destination-index/305720/6 "2022-05-30T09:38:02Z")

</div>

Thank you Sophie. I've further experimented with the sync delay value and it seems to be working as expected now

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 27, 2022, 9:38am UTC](https://discuss.elastic.co/t/latest-transform-job-not-refreshing-all-docs-in-the-destination-index/305720/7 "2022-06-27T09:38:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.