# Error while running transform \`task encountered irrecoverable failure\` **URL:** \ **Category:** Elasticsearch\ **Tags:** transforms\ **Created:** [July 3, 2024, 3:50pm UTC](https://discuss.elastic.co/t/error-while-running-transform-task-encountered-irrecoverable-failure/362479 "2024-07-03T15:50:47Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![lizozom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lizozom/32/114932_2.png) [@lizozom](https://discuss.elastic.co/u/lizozom)\ **Post date:** [July 3, 2024, 3:50pm UTC](https://discuss.elastic.co/t/error-while-running-transform-task-encountered-irrecoverable-failure/362479/1 "2024-07-03T15:50:47Z")
Hey team, I'm running ELK 8.13 I have a transform that occasionally fails to run. If I restart it - the failure persists. If I recreate it - the error goes away for a few days and that returns. It fails with this error: ```auto task encountered irrecoverable failure: [.ds-logs-stocklog-default-2024.06.05-000016/eYTuyXMETM-AvFH3-PTJyw] org.elasticsearch.index.query.QueryShardException: failed to create query: failed to parse date field [1672185600000] with format [strict_date_optional_time]: [failed to parse date field [1672185600000] with format [strict_date_optional_time]]; org.elasticsearch.ElasticsearchParseException: failed to parse date field [1672185600000] with format [strict_date_optional_time]: [failed to parse date field [1672185600000] with format [strict_date_optional_time]]; java.lang.IllegalArgumentException: failed to parse date field [1672185600000] with format [strict_date_optional_time] ``` The field that it fails on contains the values: ```auto "ref_date": "2024-04-03T00:00:00", ``` which is not a `strict_date_optional_time field`, but the transform does work if I delete and recreate the transform... Can you please help me troubleshoot and understand this error? Thanks! ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/e/8e2440c6455372dba1cfdbc6d97a0c8f58f98e53.png) ---
**Author:** ![Patrick\_Whelan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patrick_whelan/32/135049_2.png) [@Patrick\_Whelan](https://discuss.elastic.co/u/Patrick_Whelan)\ **Post date:** [July 3, 2024, 4:42pm UTC](https://discuss.elastic.co/t/error-while-running-transform-task-encountered-irrecoverable-failure/362479/2 "2024-07-03T16:42:14Z")
Are there any dates or ranges included in the source query part of the Transform config? If it's always failing on `"ref_date": "2024-04-03T00:00:00",`, it might just take time for the Transform to get to that doc once it's deleted and recreated, which would explain why it works for a bit and then fails. What is the index mapping for the `ref_date` field, and can it be changed to `strict_date_optional_time||epoch_millis` as a quick fix? > **[Get mapping API | Elasticsearch Guide \[8.14\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-mapping.html)** ---
**Author:** ![lizozom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lizozom/32/114932_2.png) [@lizozom](https://discuss.elastic.co/u/lizozom)\ **Post date:** [July 4, 2024, 6:51am UTC](https://discuss.elastic.co/t/error-while-running-transform-task-encountered-irrecoverable-failure/362479/3 "2024-07-04T06:51:17Z")
Hey Patrick The docs are never deleted \ recreated so that is probably not the issue. I also thought about changing the mapping, but if I look in the index I can't find any documents where there's a value that looks like a timestamp. Are you suggesting that `2024-04-03T00:00:00` **is** the timestamp value? 😮 ---
**Author:** ![Patrick\_Whelan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patrick_whelan/32/135049_2.png) [@Patrick\_Whelan](https://discuss.elastic.co/u/Patrick_Whelan)\ **Post date:** [July 4, 2024, 2:08pm UTC](https://discuss.elastic.co/t/error-while-running-transform-task-encountered-irrecoverable-failure/362479/4 "2024-07-04T14:08:57Z")
Kinda? Maybe? Here's what I'm thinking The error is coming from the search query - it isn't coming from the documents. Here is how I was able to reproduce it: ```auto # create index PUT /test_date_parsing # create index mapping PUT /test_date_parsing/_mapping { "properties": { "ref_date": { "type": "date", "format": "strict_date_optional_time" } } } POST /test_date_parsing/_doc { "ref_date": "2024-04-03T00:00:00" } GET /test_date_parsing/_search { "query": { "range": { "ref_date": { "gte": "1672185600000" } } } } ``` This will throw the same error, because the search query's date is in the wrong format. Transforms just runs a search request, and it will add a similar range term to the query to slice up the search requests into checkpoints. The date value will always be in `epoch_millis` (that I can see, anyway). If your Transform Config has the `sync.time.field` set to `ref_date`, then Transform will add something like `"range": { "ref_date": { "gte": "1672185600000" }` to the search request. If the source destination's mapping for `ref_date` is `strict_date_optional_time` and not `strict_date_optional_time||epoch_millis`, then the Transform will fail to perform the search request with that error. If your `sync.time.field` is not pointing to `ref_date`, then I would guess there is an additional search query term that is looking at `ref_date` and supplying an epoch value. ---
**Author:** ![lizozom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lizozom/32/114932_2.png) [@lizozom](https://discuss.elastic.co/u/lizozom)\ **Post date:** [July 10, 2024, 10:05am UTC](https://discuss.elastic.co/t/error-while-running-transform-task-encountered-irrecoverable-failure/362479/5 "2024-07-10T10:05:42Z")
Thanks @Patrick_Whelan I found the most random bug in the process. The order in which the format is specified matters. If I use `"strict_date_optional_time||epoch_millis"` it doesn't work, But if I use `"epoch_millis||strict_date_optional_time"` **it does** To test this you can create a test component template: ```auto PUT _component_template/test-logs-generic { "template": { "mappings": { "properties": { "goods.ref_date": { "type": "date", # This doesn't work "format": "strict_date_optional_time||epoch_millis", # This works # "format": "epoch_millis||strict_date_optional_time", "ignore_malformed": false } } } } } ``` Then create an index template that uses it: ```auto PUT _index_template/test-logs-template { "index_patterns": ["test-logs-*"], "priority": 1, "composed_of": ["test-logs-generic"] } ``` Then test the output with the simulate API: ```auto POST _index_template/_simulate/test-logs-template ``` You will see that the mapping differs in both cases: If `strict_date_optional_time` is first: ```auto "properties": { "ref_date": { "type": "date" } } ``` But if `epoch_millis` is first: ```auto "properties": { "ref_date": { "type": "date", "format": "epoch_millis||strict_date_optional_time" } } ``` Let me know if you agree this is an actual bug. I can open an issue (or maybe you want to pass it to the team) ---
**Author:** ![Patrick\_Whelan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patrick_whelan/32/135049_2.png) [@Patrick\_Whelan](https://discuss.elastic.co/u/Patrick_Whelan)\ **Post date:** [July 12, 2024, 4:05pm UTC](https://discuss.elastic.co/t/error-while-running-transform-task-encountered-irrecoverable-failure/362479/6 "2024-07-12T16:05:57Z")
Yeah that looks like a bug, just from the fact that the `simulate` works depending on the order. That's a great catch. Can you open an issue, and I'll try to follow up with the correct team?