# Issue with date format change while reindexing

**URL:** <https://discuss.elastic.co/t/issue-with-date-format-change-while-reindexing/169590>\
**Category:** Elasticsearch\
**Created:** [February 22, 2019, 2:00pm UTC](https://discuss.elastic.co/t/issue-with-date-format-change-while-reindexing/169590 "2019-02-22T14:00:50Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![meena](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/meena/32/71304_2.png) [@meena](https://discuss.elastic.co/u/meena)\
**Post date:** [February 22, 2019, 2:00pm UTC](https://discuss.elastic.co/t/issue-with-date-format-change-while-reindexing/169590/1 "2019-02-22T14:00:50Z")

</div>

Hi There,

Am new to elastic search and getting the following error while reindexing my data to a new index. I have a change in the format of date field from "yyyy-mm-ddThh:mm:ss.SSSZ" to "yyyy-mm-dd". Could you please help me with solving this issue

{  
"index": "index111",  
"type": "type111",  
"id": "1935",  
"cause": {  
"type": "mapper\_parsing\_exception",  
"reason": "failed to parse [person.birthDate]",  
"caused\_by": {  
"type": "illegal\_argument\_exception",  
"reason": """Invalid format: "1955-09-23T12:00:00.000Z" is malformed at "T12:00:00.000Z""""  
}  
},  
"status": 400  
}

Thanks

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 22, 2019, 2:27pm UTC](https://discuss.elastic.co/t/issue-with-date-format-change-while-reindexing/169590/2 "2019-02-22T14:27:13Z")

</div>

May be use a Date Processor that you put in an ingest pipeline.  
An ingest pipeline can be used while reindexing data.

---

<div class="post-metadata">

**Author:** ![meena](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/meena/32/71304_2.png) [@meena](https://discuss.elastic.co/u/meena)\
**Post date:** [February 22, 2019, 2:32pm UTC](https://discuss.elastic.co/t/issue-with-date-format-change-while-reindexing/169590/3 "2019-02-22T14:32:15Z")

</div>

Thanks for a quick reply. Could you please share me an example, if you have any.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 22, 2019, 2:46pm UTC](https://discuss.elastic.co/t/issue-with-date-format-change-while-reindexing/169590/4 "2019-02-22T14:46:38Z")

</div>

I don't but here is the doc:

- [https://www.elastic.co/guide/en/elasticsearch/reference/6.6/date-processor.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.6/date-processor.html)
- [https://www.elastic.co/guide/en/elasticsearch/reference/6.6/put-pipeline-api.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.6/put-pipeline-api.html)
- [https://www.elastic.co/guide/en/elasticsearch/reference/6.6/docs-reindex.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.6/docs-reindex.html)

From this last page, here is an example of calling a pipeline from reindex API:

```auto
POST _reindex
{
  "source": {
    "index": "source"
  },
  "dest": {
    "index": "dest",
    "pipeline": "some_ingest_pipeline"
  }
}

```

---

<div class="post-metadata">

**Author:** ![meena](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/meena/32/71304_2.png) [@meena](https://discuss.elastic.co/u/meena)\
**Post date:** [February 22, 2019, 3:01pm UTC](https://discuss.elastic.co/t/issue-with-date-format-change-while-reindexing/169590/5 "2019-02-22T15:01:56Z")

</div>

I have created the pipeline in the following way and tried reindex but still facing issue

PUT \_ingest/pipeline/my-pipeline-id  
{  
"description": "date pipeline ",  
"processors": [  
{  
"date": {  
"field": "person.birthDate",  
"target\_field": "timestamp",  
"formats": [  
"yyyy-MM-dd"  
]  
}  
}  
]  
}

POST \_reindex?wait\_for\_completion=false  
{  
"source": {  
"index": "source\_index"  
},  
"dest": {  
"index": "index111",  
"pipeline": "my-pipeline-id"  
}  
}

and now getting the following error

{  
"index": "index111",  
"type": "type111",  
"id": "29808",  
"cause": {  
"type": "exception",  
"reason": "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [1935-06-17T12:00:00.000Z]",  
"caused\_by": {  
"type": "illegal\_argument\_exception",  
"reason": "java.lang.IllegalArgumentException: unable to parse date [1935-06-17T12:00:00.000Z]",  
"caused\_by": {  
"type": "illegal\_argument\_exception",  
"reason": "unable to parse date [1935-06-17T12:00:00.000Z]",  
"caused\_by": {  
"type": "illegal\_argument\_exception",  
"reason": """Invalid format: "1935-06-17T12:00:00.000Z" is malformed at "T12:00:00.000Z""""  
}  
}  
},  
"header": {  
"processor\_type": "date"  
}  
},  
"status": 500  
},

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 22, 2019, 3:34pm UTC](https://discuss.elastic.co/t/issue-with-date-format-change-while-reindexing/169590/6 "2019-02-22T15:34:18Z")

</div>

Please format your code, logs or configuration files using `</>` icon as explained in [this guide](https://discuss.elastic.co/t/about-the-elasticsearch-category/21) and not the citation button. It will make your post more readable.

Or use markdown style like:

````
```
CODE
```

````

This is the icon to use if you are not using markdown format:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/e/7e6e239431ec2d71cbf1beef741f2e93e7cc762c.jpg)

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.  
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.  
Please update your post.

Here you set that you read from `person.birthDate` and write to `timestamp` but you did not change or remove `person.birthDate` which is still incorrect.

You should use the `_simulate` API to see exactly what you are doing before run the reindex operation.  
And it's probably not as straightforward as you think it is. The date processor helps to parse a string and make it a date. It does not help to render the field with the format you wish AFAIK.

But why do you want to change the date format after all? Could you explain it?  
Is that for display purpose? Can 't you solve that at render time? I mean that a date is a date in elasticsearch whatever its format in the `_source` document.

If it's about inserting data, then your field can support multiple formats. See [https://www.elastic.co/guide/en/elasticsearch/reference/master/date.html#multiple-date-formats](https://www.elastic.co/guide/en/elasticsearch/reference/master/date.html#multiple-date-formats). In which case you don't even need to reindex the data.

---

<div class="post-metadata">

**Author:** ![meena](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/meena/32/71304_2.png) [@meena](https://discuss.elastic.co/u/meena)\
**Post date:** [February 25, 2019, 10:41am UTC](https://discuss.elastic.co/t/issue-with-date-format-change-while-reindexing/169590/7 "2019-02-25T10:41:58Z")

</div>

HI David,

This is my simulate for the above issue, but i am still getting the same error, not sure if am missing something in the pipeline

```
POST _ingest/pipeline/_simulate
{
  "pipeline" :{
  "description": "date pipeline ",
  "processors": [
    {
      "date": {
        "field": "person.birthDate",
        "target_field": "timestamp",
        "formats": [
          "yyyy-MM-dd"
        ]
      }
    }
  ]},
  "docs": [
    {
      "_index": "index",
      "_type": "_doc",
      "_id": "id",
       "_source": {
          "identifiers": {
            "partyIdentifier": "20619"
          },
          "person": {
            "personIdentifier": {
              "id": "20619",
              "idScope": "ENT",
              "idContext": "B182"
            },
            "birthDate": "1950-01-01T12:00:00.000Z"}}
    }
  ]
}

```

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ERROR \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

```
        {
          "docs": [
            {
              "error": {
                "root_cause": [
                  {
                    "type": "exception",
                    "reason": "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [1950-01-01T12:00:00.000Z]",
                    "header": {
                      "processor_type": "date"
                    }
                  }
                ],
                "type": "exception",
                "reason": "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [1950-01-01T12:00:00.000Z]",
                "caused_by": {
                  "type": "illegal_argument_exception",
                  "reason": "java.lang.IllegalArgumentException: unable to parse date [1950-01-01T12:00:00.000Z]",
                  "caused_by": {
                    "type": "illegal_argument_exception",
                    "reason": "unable to parse date [1950-01-01T12:00:00.000Z]",
                    "caused_by": {
                      "type": "illegal_argument_exception",
                      "reason": """Invalid format: "1950-01-01T12:00:00.000Z" is malformed at "T12:00:00.000Z""""
                    }
                  }
                },
                "header": {
                  "processor_type": "date"
                }
              }
            }
          ]
        }
```

---

<div class="post-metadata">

**Author:** ![cinhtau](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cinhtau/32/40292_2.png) [@cinhtau](https://discuss.elastic.co/u/cinhtau)\
**Post date:** [February 25, 2019, 11:15am UTC](https://discuss.elastic.co/t/issue-with-date-format-change-while-reindexing/169590/8 "2019-02-25T11:15:02Z")

</div>

> [@meena](#):
>
> "formats": ["yyyy-MM-dd"]

Use `ISO8601` as the format.

```auto
"formats": ["ISO8601"]

```

Your input doesn't match your format.

```auto
Invalid format: "1950-01-01T12:00:00.000Z"

```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 25, 2019, 12:04pm UTC](https://discuss.elastic.co/t/issue-with-date-format-change-while-reindexing/169590/9 "2019-02-25T12:04:14Z")

</div>

Can you also answer my questions?

> But why do you want to change the date format after all? Could you explain it?  
> Is that for display purpose? Can 't you solve that at render time? I mean that a date is a date in elasticsearch whatever its format in the `_source` document.

---

<div class="post-metadata">

**Author:** ![meena](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/meena/32/71304_2.png) [@meena](https://discuss.elastic.co/u/meena)\
**Post date:** [February 25, 2019, 3:43pm UTC](https://discuss.elastic.co/t/issue-with-date-format-change-while-reindexing/169590/10 "2019-02-25T15:43:46Z")

</div>

The source and destination are in yyyy-mm-dd format so to have a uniform data format we are looking for this change. we do have some other fields with changes in mappings and settings so we would like to do all the changes in one shot by doing reindexing

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 26, 2019, 5:33pm UTC](https://discuss.elastic.co/t/issue-with-date-format-change-while-reindexing/169590/11 "2019-02-26T17:33:54Z")

</div>

Here is a way to do it:

```auto
POST _ingest/pipeline/_simulate
{
  "pipeline" :{
  "description": "date pipeline ",
  "processors": [
    {
        "script": {
          "source": """
             SimpleDateFormat format = new SimpleDateFormat("yyyy-MM-dd");
             ctx.person.birthDate = format.format(format.parse(ctx.person.birthDate));
          """
        }
    }
  ]},
  "docs": [
    {
      "_index": "index",
      "_type": "_doc",
      "_id": "id",
       "_source": {
          "person": {
            "birthDate": "1950-01-01T12:00:00.000Z"
          }
       }
    }
  ]
}

```

This gives:

```auto
{
  "docs" : [
    {
      "doc" : {
        "_index" : "index",
        "_type" : "_doc",
        "_id" : "id",
        "_source" : {
          "person" : {
            "birthDate" : "1950-01-01"
          }
        },
        "_ingest" : {
          "timestamp" : "2019-02-26T17:35:45.52603Z"
        }
      }
    }
  ]
}

```

HTH

---

<div class="post-metadata">

**Author:** ![meena](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/meena/32/71304_2.png) [@meena](https://discuss.elastic.co/u/meena)\
**Post date:** [February 28, 2019, 9:12am UTC](https://discuss.elastic.co/t/issue-with-date-format-change-while-reindexing/169590/12 "2019-02-28T09:12:04Z")

</div>

Thank you David, Thats very helpful 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 28, 2019, 9:12am UTC](https://discuss.elastic.co/t/issue-with-date-format-change-while-reindexing/169590/13 "2019-03-28T09:12:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.