Transform is not created when a particular field is not available in the index

[Amol (Amol Barbade)]

Hello Team,

I'm trying to create a transform, it is creating the transform index, but when the particular (Year column) is blank or not available in the Data set document, that document is not index.

Could any one help in the below transform json query!

PUT _transform/year-test1
{
  "source": {
    "index": [
      "newyear22"
    ]
  },
  "pivot": {
    "group_by": {
      "usrid.keyword": {
        "terms": {
          "field": "usrid.keyword"
        }
      },
      "dept.keyword": {
        "terms": {
          "field": "dept.keyword"
        }
      },
      "year.keyword": {
        "terms": {
          "field": "year.keyword"
        }
      }
    },
    "aggregations": {
      "activity.keyword.cardinality": {
        "cardinality": {
          "field": "activity.keyword"
        }
      }
    }
  },
  "description": "Transform testing Year null values",
  "frequency": "1m",
  "dest": {
    "index": "year-test1"
  },
  "sync": {
    "time": {
      "field": "@timestamp",
      "delay": "60s"
    }
  },
  "settings": {
    "max_page_search_size": 500
  }
}

Many thanks in advance!!
@Amol

[sophie_chang]

Assuming the field in the document is sometimes blank, it sounds like you will need to get a bit more creative with your aggregation.

Support for the missing aggregation was added in 7.11. Alternatively you may require a runtime field, or you could resolve the missing data at ingest.

This is not configurable via the UI, so you will need to use Dev Tools to create the transform. (When editing in Dev Tools, suggest using _preview as a way to more easily validate the syntax.)

[Amol (Amol Barbade)]

Thank you @sophie_chang for your response.

It worked for me by adding one more lines to the fields section.

"missing_bucket": true