Hello!
I'm trying to create a transform from two different indices but keep on running to this error and have tried many different solutions but can't get it to work.
This is the error:
Failed to index documents into destination index due to permanent error
[BulkIndexingException[Bulk index experienced [160] failures and at least 1 irrecoverable [TransformException[Destination index mappings are incompatible with the transform configuration.]; nested: MapperParsingException[failed to parse field [user_created] of type [date] in document with id '<doc_id>'. Preview of field's value: '1.513493852792E12']; nested: IllegalArgumentException[failed to parse date field [1.513493852792E12] with format [strict_date_optional_time||epoch_millis]]; nested: DateTimeParseException[Failed to parse with all enclosed parsers];
This is the transform I'm running:
{
"source": {
"index": [
"<first_index>",
"<second_index>"
]
},
"dest": {
"index": "<combined_index>"
},
"pivot": {
"group_by": {
"user_id": {
"terms": {
"field": "user_id"
}
}
},
"aggregations": {
"user_created": {
"max": {
"field": "user_created"
}
},
"last_access": {
"max": {
"field": "last_access"
}
}
}
}
}
And this is an example of the data I get running the preview:
{
"last_access" : 1.513493852792E12,
"user_id" : "<user_id_string>",
"user_created" : "2015-01-24T12:33:18.263Z"
},
In some cases the user_created
date is also the same format as the last_access
in this example.
And this is the mapping in the destination index:
{
"properties": {
"user_id": {
"type": "keyword"
},
"user_created": {
"type": "date"
},
"last_access": {
"type": "date"
}
}
}
In both of the indices the dates are completely normal "yyyy-MM-dd'T'HH:mm:ss.SSSZ"
format but something happens in the transform that changes it to this 1.513493852792E12
kind of format that causes the error. Kind of still figuring out transforms and elastic as a whole.
Is there something that I'm missing or what could cause this kind of behavior? Thanks a lot!