Error reindex some data

Hi,

I've been trying to migrate my ELK from v6.8 to v7.10, but prior to this i have to solve some issues pointed by 7.0 Upgrade Assistant.

I have some fields with the format: yyyy-MM-dd HH:mm:ss, i've already created a paralel index with new date format: "formats" : ["8uuuu/MM/dd HH:mm:ss"], have altered also the ingest pipeline to be in the new date format.
I've followed this recomendations: https://www.elastic.co/guide/en/elasticsearch/reference/7.x/migrate-to-java-time.html

When i try to reindex the data, i'm getting this error:

{
  "took" : 25,
  "timed_out" : false,
  "total" : 11,
  "updated" : 0,
  "created" : 0,
  "deleted" : 0,
  "batches" : 1,
  "version_conflicts" : 0,
  "noops" : 0,
  "retries" : {
    "bulk" : 0,
    "search" : 0
  },
  "throttled_millis" : 0,
  "requests_per_second" : -1.0,
  "throttled_until_millis" : 0,
  "failures" : [
    {
      "index" : "pacienteunificado2",
      "type" : "pacienteunificado",
      "id" : "96408",
      "cause" : {
        "type" : "mapper_parsing_exception",
        "reason" : "failed to parse field [dataAtualizacao] of type [date] in document with id '96408'",
        "caused_by" : {
          "type" : "illegal_argument_exception",
          "reason" : "failed to parse date field [2020-01-24 03:15:13] with format [8uuuu/MM/dd HH:mm:ss]",
          "caused_by" : {
            "type" : "date_time_parse_exception",
            "reason" : "Text '2020-01-24 03:15:13' could not be parsed at index 4"
          }
        }
      },
      "status" : 400
    },

Is there anything that i've been missing, in order to do this?

Adding some information:
Here is my new index with some datetime fields

        "dataAtualizacao": {
          "type": "date",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "format": "8uuuu/MM/dd HH:mm:ss"
        },
        "dataCadastro": {
          "type": "date",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "format": "8uuuu/MM/dd HH:mm:ss"
        },
        "dataNascimento": {
          "type": "date",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "format": "8uuuu/MM/dd HH:mm:ss"
        },
        "dataObito": {
          "type": "date",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          },
          "format": "8uuuu/MM/dd HH:mm:ss"
        },

and here is the ingest-pipeline:

{
  "pacienteunificado-pipeline" : {
    "description" : "Ingest pipeline created by file structure finder",
    "processors" : [
      {
        "grok" : {
          "field" : "message",
          "patterns" : [
            "          %{QUOTEDSTRING:field} : \"%{TIMESTAMP_ISO8601:timestamp}\",\\n          %{QUOTEDSTRING:field2} : %{QUOTEDSTRING:field3},.*"
          ]
        }
      },
      {
        "date" : {
          "field" : "timestamp",
          "timezone" : "America/Sao_Paulo",
          "formats" : [
            "8uuuu/MM/dd HH:mm:ss"
          ]
        }
      },
      {
        "remove" : {
          "field" : "timestamp"
        }
      }
    ]
  }
}

The error reported on the post above, is related with the reindex from the old index to this new one.

Any help or advise on how to solve this?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.