Elastic transform

[Amol (Amol Barbade)]

Hello Team,

I'm able to create index through Transform with the help of Dev tool.
But to avoid or over-write documents with same unique ID, I need to pass Document ID in transform, but I'm not able to add it.

Kindly anyone could suggest, how to include document ID in transform through Dev tool in kibana.

PUT _transform/unique_user_jul22
{
  "source": {
    "index": [
      "test_index"
    ],
    "query": {
      "bool": {
        "filter": [
          {
            "range": {
              "@timestamp": {
                "gte": "2022-07-01T00:00:00.000Z",
                "lte": "2022-07-31T23:59:59.920Z"
              }
            }
          }
        ]
      }
    }
  },
  "pivot": {
    "group_by": {
      "name.keyword": {
        "terms": {
          "field": "name.keyword"
        }
      }
    },
    "aggregations": {
      "active.keyword.cardinality": {
        "cardinality": {
          "field": "active.keyword"
        }
      }
    }
  },
  "frequency": "1m",
  "dest": {
    "index": "user_jul22"
  },
  "sync": {
    "time": {
      "field": "@timestamp",
      "delay": "60s"
    }
  },
  "settings": {
    "max_page_search_size": 500
  }
}

With Regards,
@Amol

[Hendrik_Muhs]

By design you can't overwrite the doc id in a transform. However it is possible using ingest:

You can overwrite the doc id if you attach an ingest pipeline and overwrite it there with a set processor or you generate a new document id with a fingerprint processor.

I wonder: You can add a date_histogram in the group_by and create 1 bucket per minute per user without manipulating the doc id.