# Latest Transform questions

**URL:** <https://discuss.elastic.co/t/latest-transform-questions/297293>\
**Category:** Elasticsearch\
**Tags:** transforms\
**Created:** [February 15, 2022, 8:19pm UTC](https://discuss.elastic.co/t/latest-transform-questions/297293 "2022-02-15T20:19:26Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![iamtheschmitzer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iamtheschmitzer/32/96988_2.png) [@iamtheschmitzer](https://discuss.elastic.co/u/iamtheschmitzer)\
**Post date:** [February 15, 2022, 8:19pm UTC](https://discuss.elastic.co/t/latest-transform-questions/297293/1 "2022-02-15T20:19:26Z")

</div>

Creating a latest transform using the API, I need to provide a set of unique\_keys, which can be a list of field names.

Some questions:

1. How do I reference a field in a sub object? Like within `foo` there is a key `bar` that needs to be part of my uniqueness set?
2. Is there a way to select all fields EXCEPT a list (an exclude list)?
3. Can two transforms share a destination index
4. Is there a way to specify ALL fields in a sub object? Like `foo.*`
5. How do you specify fields within an array? Where `foo` is an array of objects with `bar` inside?

All these are related in my mind. I need 1, would like to use 2 and if not I need 3 or 4+

Thanks in Advance

---

<div class="post-metadata">

**Author:** ![iamtheschmitzer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iamtheschmitzer/32/96988_2.png) [@iamtheschmitzer](https://discuss.elastic.co/u/iamtheschmitzer)\
**Post date:** [February 16, 2022, 1:06am UTC](https://discuss.elastic.co/t/latest-transform-questions/297293/2 "2022-02-16T01:06:02Z")

</div>

#1 solved, it accepts dot syntax, so `foo.bar` finds my field.

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 16, 2022, 5:08am UTC](https://discuss.elastic.co/t/latest-transform-questions/297293/3 "2022-02-16T05:08:17Z")

</div>

1. 

Where do you want to exclude fields? I suppose there is no settings about fields of latest transform.  
That said, it could be realized [remove ingest pipline](https://www.elastic.co/guide/en/elasticsearch/reference/current/remove-processor.html) of the dest index.

1. 

Yes but if `unique_key` is common between source indices, dest index could be updated by the older index. Using index pattern or array of indecies in a single transform could be recommended.

---

<div class="post-metadata">

**Author:** ![iamtheschmitzer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iamtheschmitzer/32/96988_2.png) [@iamtheschmitzer](https://discuss.elastic.co/u/iamtheschmitzer)\
**Post date:** [February 16, 2022, 1:23pm UTC](https://discuss.elastic.co/t/latest-transform-questions/297293/4 "2022-02-16T13:23:32Z")

</div>

Thanks I want all fields except one in `unique_key`. So for everything but timestamp, I want to express this:

`"unique_key": "!timestamp"`

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 16, 2022, 2:34pm UTC](https://discuss.elastic.co/t/latest-transform-questions/297293/5 "2022-02-16T14:34:53Z")

</div>

I tested some and found a TRICK. (Remember it is just a trick.)

Here is the setting.

```auto
PUT /test_unique_key/
{
  "mappings": {
    "properties": {
      "id":{"type":"keyword"},
      "date":{"type":"date"}
    }
  }
}

POST /test_unique_key/_bulk
{"index":{}}
{"id":1,"date":"2011-10-10T00:00:00.000Z"}
{"index":{}}
{"id":1,"date":"2020-10-10T00:00:00.000Z"}

```

If you set "timestamp" ("date" here) explicitly

```auto
GET /_transform/_preview?filter_path=preview
{
  "latest":{
    "sort":"date",
    "unique_key": ["*","date"]
  },
  "dest": "test_dest",
  "source":{
    "index":"test_unique_key"
  }
}

```

Two documents are distinguished.

```auto
{
  "preview" : [
    {
      "date" : "2011-10-10T00:00:00.000Z",
      "id" : 1
    },
    {
      "date" : "2020-10-10T00:00:00.000Z",
      "id" : 1
    }
  ]
}

```

If you set wildcard containing "timestamp",:

```auto
GET /_transform/_preview?filter_path=preview
{
  "latest":{
    "sort":"date",
    "unique_key": ["*"]
  },
  "dest": "test_dest",
  "source":{
    "index":"test_unique_key"
  }
}

```

The difference between "timestamp" fields are ignored and there are only one document.

```auto
{
  "preview" : [
    {
      "date" : "2020-10-10T00:00:00.000Z",
      "id" : 1
    }
  ]
}

```

This could be a solution. As this behavior is not explained in [the official document](https://www.elastic.co/guide/en/elasticsearch/reference/current/preview-transform.html), however, I'm not sure this is just for this specific case, for example it is not likely but it could work only with "date" field name..., or works for general situations. And also it could dissapear or change suddenly without any notion.

If you want to use this, please do thorough testing and use with extreme caution.

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 16, 2022, 2:36pm UTC](https://discuss.elastic.co/t/latest-transform-questions/297293/6 "2022-02-16T14:36:25Z")

</div>

IMO, if you want to deduplicate the same documents, using ingest pipeline with [Fingerprint processor](https://www.elastic.co/guide/en/elasticsearch/reference/master/fingerprint-processor.html) during indexing time could be a better solution than latest transform.

---

<div class="post-metadata">

**Author:** ![iamtheschmitzer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iamtheschmitzer/32/96988_2.png) [@iamtheschmitzer](https://discuss.elastic.co/u/iamtheschmitzer)\
**Post date:** [February 16, 2022, 5:25pm UTC](https://discuss.elastic.co/t/latest-transform-questions/297293/7 "2022-02-16T17:25:06Z")

</div>

This solves my problem, I think. Not the "all fields except" uniqueness problem exactly, but it lets me get around it by solving 4 above. All fields in sub-object `foo` are the equivalent of `foo`'s fingerprint. Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 16, 2022, 5:25pm UTC](https://discuss.elastic.co/t/latest-transform-questions/297293/8 "2022-03-16T17:25:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.