Transform invalid date values

[liorg2]

Hello,

I have the following group by in my transform:

"pivot": {
    "group_by": {

      "datetime": {
        "date_histogram": {
          "field": "datetime",
          "calendar_interval": "1m"
        }
      },

when checking the max date value in the source index I get:

GET logs/_search?filter_path=hits.hits._source
{
  "size": 1, 
  "_source": "datetime", 
  "sort": [
    {
      "datetime": {
        "order": "desc"
      }
    }
  ]
}

{
  "hits" : {
    "hits" : [
      {
        "_source" : {
          "datetime" : "2020-09-21T06:36:34.000Z"
        }
      }
    ]
  }
}

when checking the transform's result index using the same query body I get:

{
  "hits" : {
    "hits" : [
      {
        "_source" : {
          "datetime" : 1602181620000
        }
      }
    ]
  }
}

which is Thursday, October 8, 2020 6:27:00 PM GMT

anything wrong that I'm doing?

will it affect the continues mode checkpoint ?

Thanks

[Hendrik_Muhs]

This is hard to answer as their could be many reasons:

• broken mappings in the source (return doc values or use an aggregation in your search)
• delay configured in transform (default 60s, you could put a range filter in your search query to emulate the same behavior)
• transform hasn't processed the data yet (check _stats regarding checkpoints)

To better help you, I think you should use support, so they can run diagnostics.