Error reindex some data

Elastic Stack Elasticsearch
,
1

Hi,

I've been trying to migrate my ELK from v6.8 to v7.10, but prior to this i have to solve some issues pointed by 7.0 Upgrade Assistant.

I have some fields with the format: yyyy-MM-dd HH:mm:ss, i've already created a paralel index with new date format: "formats" : ["8uuuu/MM/dd HH:mm:ss"], have altered also the ingest pipeline to be in the new date format.
I've followed this recomendations: https://www.elastic.co/guide/en/elasticsearch/reference/7.x/migrate-to-java-time.html

When i try to reindex the data, i'm getting this error:

{
  "took" : 25,
  "timed_out" : false,
  "total" : 11,
  "updated" : 0,
  "created" : 0,
  "deleted" : 0,
  "batches" : 1,
  "version_conflicts" : 0,
  "noops" : 0,
  "retries" : {
    "bulk" : 0,
    "search" : 0
  },
  "throttled_millis" : 0,
  "requests_per_second" : -1.0,
  "throttled_until_millis" : 0,
  "failures" : [
    {
      "index" : "pacienteunificado2",
      "type" : "pacienteunificado",
      "id" : "96408",
      "cause" : {
        "type" : "mapper_parsing_exception",
        "reason" : "failed to parse field [dataAtualizacao] of type [date] in document with id '96408'",
        "caused_by" : {
          "type" : "illegal_argument_exception",
          "reason" : "failed to parse date field [2020-01-24 03:15:13] with format [8uuuu/MM/dd HH:mm:ss]",
          "caused_by" : {
            "type" : "date_time_parse_exception",
            "reason" : "Text '2020-01-24 03:15:13' could not be parsed at index 4"
          }
        }
      },
      "status" : 400
    },

Is there anything that i've been missing, in order to do this?

2

Adding some information:
Here is my new index with some datetime fields

        "dataAtualizacao": {
          "type": "date",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "format": "8uuuu/MM/dd HH:mm:ss"
        },
        "dataCadastro": {
          "type": "date",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "format": "8uuuu/MM/dd HH:mm:ss"
        },
        "dataNascimento": {
          "type": "date",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "format": "8uuuu/MM/dd HH:mm:ss"
        },
        "dataObito": {
          "type": "date",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "format": "8uuuu/MM/dd HH:mm:ss"
        },

and here is the ingest-pipeline:

{
  "pacienteunificado-pipeline" : {
    "description" : "Ingest pipeline created by file structure finder",
    "processors" : [
      {
        "grok" : {
          "field" : "message",
          "patterns" : [
            "          %{QUOTEDSTRING:field} : \"%{TIMESTAMP_ISO8601:timestamp}\",\\n          %{QUOTEDSTRING:field2} : %{QUOTEDSTRING:field3},.*"
          ]
        }
      },
      {
        "date" : {
          "field" : "timestamp",
          "timezone" : "America/Sao_Paulo",
          "formats" : [
            "8uuuu/MM/dd HH:mm:ss"
          ]
        }
      },
      {
        "remove" : {
          "field" : "timestamp"
        }
      }
    ]
  }
}

The error reported on the post above, is related with the reindex from the old index to this new one.

Any help or advise on how to solve this?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.