# Transform is only partially updated **URL:** \ **Category:** Elasticsearch\ **Tags:** transforms\ **Created:** [January 28, 2024, 9:30am UTC](https://discuss.elastic.co/t/transform-is-only-partially-updated/351935 "2024-01-28T09:30:13Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Doron\_Abramovich](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doron_abramovich/32/131186_2.png) [@Doron\_Abramovich](https://discuss.elastic.co/u/Doron_Abramovich)\ **Post date:** [January 28, 2024, 9:30am UTC](https://discuss.elastic.co/t/transform-is-only-partially-updated/351935/1 "2024-01-28T09:30:13Z")
Hi everyone, I have an transform that suppose to track the latest doc of some index. The transform I made is based on field called "etl\_id" and for some reason it is updated only for few "etl\_id" but not all of them, here is an example of some doc that was not updated in the transform: source index ( **etl\_logs** ): ```auto { "_index": "etl_logs", "_id": "C1FKPI0BOAhJ8yM6psQ7", "_score": null, "_source": { "level": "INFO", "host": "host1", "etl_filename": "etl.py", "etl_id": "1988", "items_processed": 1, "log_data": "", "timestamp": "2024-01-24T18:25:41+03:00" } ``` transform index ( **latest\_etl\_log** ): ```auto { "_index": "latest_etl_log", "_id": "MYWbD3aM_AEJlF6WStE10ckAAAAAAAAA", "_score": null, "_source": { "items_processed": 1, "etl_id": "1988", "level": "INFO", "etl_filename": "etl.py", "host": "host1", "log_data": "", "timestamp": "2024-01-22T16:52:57+03:00" } ``` And here is the transform settings: ```auto { "count": 1, "transforms": [ { "id": "latest_etl_log", "authorization": { "roles": [ "superuser" ] }, "version": "8.7.1", "create_time": 1705401981779, "source": { "index": [ "etl_logs" ], "query": { "match_all": {} } }, "dest": { "index": "latest_etl_log" }, "sync": { "time": { "field": "timestamp", "delay": "60s" } }, "latest": { "unique_key": [ "etl_id.keyword" ], "sort": "timestamp" }, "settings": {} } ] } ``` Any ideas what can be the problem? why some of the etl\_id are updated correctly and the others stays behind? ---
**Author:** ![greco](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/greco/32/106815_2.png) [@greco](https://discuss.elastic.co/u/greco)\ **Post date:** [January 29, 2024, 7:59am UTC](https://discuss.elastic.co/t/transform-is-only-partially-updated/351935/2 "2024-01-29T07:59:45Z")
Hi @Doron_Abramovich, Can you please try changing your transform configuration and use ```auto "unique_key": [ "etl_id" ], ``` instead ? That being said, I'm not sure I fully understand your problem : are you missing latest docs for some _etl\_id_, or do you have docs in your destination index for every _etl\_id_, but these docs are not the latest ones ? The example you provided was describing the latter, and it's a different problem overall. ---
**Author:** ![Doron\_Abramovich](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doron_abramovich/32/131186_2.png) [@Doron\_Abramovich](https://discuss.elastic.co/u/Doron_Abramovich)\ **Post date:** [January 29, 2024, 9:49am UTC](https://discuss.elastic.co/t/transform-is-only-partially-updated/351935/3 "2024-01-29T09:49:02Z")
@greco Yeah sorry it was not so clear, I have all needed **etl\_id** on the destination index but only some of them are keep on updating, the example I gave was a document that was suppose to be updated according to the source index (with the date **2024-01-24** ) where the destination index after transformed holding older document (with the date **2024-01-22** ) I will try your solution anyhow and let you know if it works out 🙂 **edited:** while trying to use **etl\_id** instead if **etl\_id.keyword** I get an error message ```auto Fielddata is disabled on [etl_id] in [etl_logs]. Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. ``` Is there another solution you can think of, You have an idea of what might be the problem? ---
**Author:** ![greco](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/greco/32/106815_2.png) [@greco](https://discuss.elastic.co/u/greco)\ **Post date:** [January 29, 2024, 3:39pm UTC](https://discuss.elastic.co/t/transform-is-only-partially-updated/351935/4 "2024-01-29T15:39:32Z")
Ok, thanks for clarifying. The only reason I can think of for these missing docs would be a transform that struggles to catch up for some values of etl\_id : do you have a lot of documents ? Are your transforms lagging ? ---
**Author:** ![Doron\_Abramovich](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doron_abramovich/32/131186_2.png) [@Doron\_Abramovich](https://discuss.elastic.co/u/Doron_Abramovich)\ **Post date:** [January 30, 2024, 9:34am UTC](https://discuss.elastic.co/t/transform-is-only-partially-updated/351935/5 "2024-01-30T09:34:58Z")
There are only 45 types of **etl\_id** options, thats pretty solid i guess.. The **etl\_id** can be both a number or string, for exmaple - "1424" and "FTP\_daily\_process" are two different types. The field mapping of source index **etl\_logs** is : ```auto "etl_id": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } ``` I now changed it to be a **fieldata** and use **etl\_id** instead if **etl\_id.keyword** , will let you know soon if it worked out ---
**Author:** ![Doron\_Abramovich](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doron_abramovich/32/131186_2.png) [@Doron\_Abramovich](https://discuss.elastic.co/u/Doron_Abramovich)\ **Post date:** [January 30, 2024, 12:34pm UTC](https://discuss.elastic.co/t/transform-is-only-partially-updated/351935/6 "2024-01-30T12:34:53Z")
**Update:** The problem still happens, my other transforms works properly, it is just this one.. ---
**Author:** ![greco](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/greco/32/106815_2.png) [@greco](https://discuss.elastic.co/u/greco)\ **Post date:** [January 30, 2024, 12:48pm UTC](https://discuss.elastic.co/t/transform-is-only-partially-updated/351935/8 "2024-01-30T12:48:59Z")
Thank you for trying all this, We need a bit more time to investigate further. ---
**Author:** ![Doron\_Abramovich](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doron_abramovich/32/131186_2.png) [@Doron\_Abramovich](https://discuss.elastic.co/u/Doron_Abramovich)\ **Post date:** [January 30, 2024, 2:10pm UTC](https://discuss.elastic.co/t/transform-is-only-partially-updated/351935/9 "2024-01-30T14:10:15Z")
After quick check on the transform messages I can see that I do get a warning message: `Non-empty destination index [latest_etl_log]. Contains [70] total documents.` Thank you! I appreciate your help 🙂 ---
**Author:** ![przemekwitek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/przemekwitek/32/79526_2.png) [@przemekwitek](https://discuss.elastic.co/u/przemekwitek)\ **Post date:** [January 30, 2024, 2:39pm UTC](https://discuss.elastic.co/t/transform-is-only-partially-updated/351935/10 "2024-01-30T14:39:24Z")
One more reason for missing updates I can think of is that when the document is ingested into source index, its timestamp is "old", i.e.: older than `60s` (the configured delay) than the actual server timestamp. Usually the solution for that is to have an ingest pipeline on the **source** index that will populate `event.ingested` field for every source document and then to use `event.ingested` field in the `sync.time` section of the transform config. You can find the pipeline code here: > **[How transform checkpoints work | Elasticsearch Guide \[8.12\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-checkpoints.html#sync-field-ingest-timestamp)** ---
**Author:** ![Doron\_Abramovich](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/doron_abramovich/32/131186_2.png) [@Doron\_Abramovich](https://discuss.elastic.co/u/Doron_Abramovich)\ **Post date:** [January 31, 2024, 9:54am UTC](https://discuss.elastic.co/t/transform-is-only-partially-updated/351935/11 "2024-01-31T09:54:19Z")
This is actually a good point, As I remember I struggled using the server timestamp for something because it was not match to my country timezone, The only question is.. How come this problem happens only for a certain index? Maybe it happens because this time field name in this index is "timestamp" which is the same as the server uses? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 28, 2024, 9:54am UTC](https://discuss.elastic.co/t/transform-is-only-partially-updated/351935/12 "2024-02-28T09:54:20Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.