Reindex string to date field exception

Elastic Stack Elasticsearch
1

Hello,
I've done much searching on the topics of converting a field that's defined or mapped as "text string" to type date but can't find a specific example like what I'm trying. Any help would be appreciated.

This index is created from filebeat input (log file) to logstash. And, I can't find the original index fields mapping. The problem is a date field has type "text string" that I want to be type date for the purposes of date range querying.

Because the type of a field can't be changed, I'm trying to create a new index with all the fields the same as the current index. With two fields changed to have type date. Then, I tried to run _reindex to copy data from current index to new index.

The new index has all the data okay but the two fields that I defined as date type still have type "text string". That's because reindex hit the following exceptions:

{
  "took": 276,
  "timed_out": false,
  "total": 252,
  "updated": 0,
  "created": 0,
  "deleted": 0,
  "batches": 1,
  "version_conflicts": 0,
  "noops": 0,
  "retries": {
    "bulk": 0,
    "search": 0
  },
  "throttled_millis": 0,
  "requests_per_second": -1,
  "throttled_until_millis": 0,
  "failures": [
    {
      "index": "cast-allocation-test",
      "type": "_doc",
      "id": "193",
      "cause": {
        "type": "mapper_parsing_exception",
        "reason": "failed to parse field [data.begin_time] of type [date] in document with id '193'. Preview of field's value: '2020-11-02 13:04:22.413672'",
        "caused_by": {
          "type": "illegal_argument_exception",
          "reason": "failed to parse date field [2020-11-02 13:04:22.413672] with format [YYYY-MM-dd HH:mm:ss.000000]",
          "caused_by": {
            "type": "date_time_parse_exception",
            "reason": "Text '2020-11-02 13:04:22.413672' could not be parsed at index 20"
          }
        }
      },
      "status": 400
    },

From the exception above, you can see the original text string looks like this:

'2020-11-02 13:04:22.413672'

And, my new index date type format for those two fields looks like:

YYYY-MM-dd HH:mm:ss.000000

I'd like to check if the format and what is done is the right way to do this. Thanks in advance for any help.

My reindex command:

POST _reindex
{
  "source": {
    "index": "cast-allocation"
  },
  "dest": {
    "index": "cast-allocation-test"
  }
}
2

Hmm.... Nobody found a problem here?

After more try and error experiments, I kind of concluding that the problem is in the text string of date format having a blank between date and time:

'2020-11-02 13:04:22.413672'

Can't even write that string to a new record with a date field. So this is probably the problem with reindex also. It didn't matter what date format was defined for the date field.

Using the same text string with the letter T replacing the blank, I can write the new record with no problem. I also removed the format for the new date field. That makes the format more flexible.

Back to the problem with reindex, I wonder if a "processing" "script" can be added to reindex to add the T into the text string date before the record being written into the new index? Something like:

POST _reindex
{
  "source": {
    "index": "cast-allocation"
  },
  "dest": {
    "script": "<change 'date time' to 'dateTtime'>"
    "index": "cast-allocation-test"
  }
}

Can someone help with an example of the above "script"?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.