Transform date field (timeStamp) "failed to parse field"

[Cain310 (Michael Cain)]

Hi I am having trouble with transforming a timeStamp date field for ml.

I am getting this error in the logs

failed to parse field [doc.timeStamp.min] of type [date] in document with id 'xxx'.
Preview of field's value: '2019-11-28 09:35:05'

Ive tried creating a mapping dynamic template and setting this fields format prior to running the transform

"dynamic_templates": [
      {
        "dates": {
          "match": ".*timeStamp*",
          "match_pattern": "regex",
          "mapping": {
            "format": "yyyy-MM-dd HH:mm:ss||dateOptionalTime||MM/dd/yyyy||yyyy-MM-dd",
            "ignore_malformed": true,
            "type": "date"
          }
        }
      }
    ]

As well as attempting to set the format in advanced pivot editor.

"doc.timeStamp.min": {
        "min": {
          "field": "doc.timeStamp",
          "format": "yyyy-MM-dd HH:mm:ss||dateOptionalTime||MM/dd/yyyy||yyyy-MM-dd"
        }
      },

stumped..

[Hendrik_Muhs]

This sounds like a bug to me. For reproduction it would be good to have a full data sample: Can you post an example document, at least an example for doc.timeStamp? Which version are you on?

Transform deducts mappings from the source and creates a mapping according to that. You can see the deduced mappings in the output of _preview. Please run _preview in dev console, you can copy the command from the UI using the upper right icon in the preview table.

For customizing mappings, create the index yourself, index templates are not taken into account. You can use the output of _preview for this.

[Cain310 (Michael Cain)]

Thanks for the help Hendrik,

I created the mapping manually before hand using the preview and was able to set the date format manually in dev tools. Probably a bug with the transform ui as the source doc.timeStamp I'm pulling from was the same. Also the transform preview shows the date parsed correctly prior to begining the transform which is strange. I will test my ml jobs hopefully they work the same.

For reference i am using 7.6.2

Source doc.timeStamp mapping

"timeStamp" : {
              "type" : "date",
              "ignore_malformed" : true,
              "format" : "yyyy-MM-dd HH:mm:ss||dateOptionalTime||MM/dd/yyyy||yyyy-MM-dd"
            },

I also noticed the resulting mapping of the transform now looks different in comparision to when i did not set the mapping manually.

{
  "xxx" : {
    "mappings" : {
      "dynamic_templates" : [
        {
          "dates" : {
            "match" : ".*timeStamp*",
            "match_pattern" : "regex",
            "mapping" : {
              "format" : "yyyy-MM-dd HH:mm:ss||dateOptionalTime||MM/dd/yyyy||yyyy-MM-dd",
              "ignore_malformed" : true,
              "type" : "date"
            }
          }
        }
      ],
      "properties" : {
        "xxx" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "xxx" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "xxx" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "xxx" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "xxx" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "xxx" : {
          "type" : "double"
        },
        "xxx" : {
          "type" : "long"
        },
        "xxx" : {
          "type" : "long"
        },
        "doc" : {
          "properties" : {
            "xxx" : {
              "properties" : {
                "keyword" : {
                  "type" : "keyword"
                }
              }
            },
            "timeStamp" : {
              "properties" : {
                "max" : {
                  "type" : "date",
                  "ignore_malformed" : true,
                  "format" : "yyyy-MM-dd HH:mm:ss||dateOptionalTime||MM/dd/yyyy||yyyy-MM-dd"
                },
                "min" : {
                  "type" : "date",
                  "ignore_malformed" : true,
                  "format" : "yyyy-MM-dd HH:mm:ss||dateOptionalTime||MM/dd/yyyy||yyyy-MM-dd"
                }
              }
            }
          }
        },
        "xxx" : {
          "properties" : {
            "sum" : {
              "type" : "double"
            }
          }
        }
      }
    }
  }
}

Where before it omitted any scripted field mappings. Not sure if this is purposefully done.

{
  "xxx" : {
    "mappings" : {
      "_meta" : {
        "created_by" : "transform",
        "_transform" : {
          "transform" : "xxx",
          "version" : {
            "created" : "7.6.2"
          },
          "creation_date_in_millis" : 1586896729706
        }
      },
      "properties" : {
        "xxx" : {
          "type" : "double"
        },
        "xxx" : {
          "type" : "long"
        },
        "xxx" : {
          "type" : "long"
        },
        "doc" : {
          "properties" : {
            "xxx" : {
              "properties" : {
                "keyword" : {
                  "type" : "keyword"
                }
              }
            },
            "timeStamp" : {
              "properties" : {
                "max" : {
                  "type" : "date"
                },
                "min" : {
                  "type" : "date"
                }
              }
            }
          }
        },
        "xxx" : {
          "properties" : {
            "sum" : {
              "type" : "double"
            }
          }
        }
      }
    }
  }
}