Reindex - total vs. created

Elastic Stack Elasticsearch
1

Hi helpers,

I have some field mapping conflicts on one of my indices and would like to reindex it.

So I printed out the correct index, copyed/corrected the mapping structure, created a new index and tried to reindex data from old to new index. The problem is, not all data is reindexed. What am I doing wrong?

The printout:

curl -XPOST 'localhost:9200/_reindex?&pretty' -d '{
>  "conflicts": "proceed",
>  "source" : {
>   "index" : "mmsc-2017.06.05"
>  },
>  "dest" : {
>   "index" : "mmsc-2017.06.05_new",
>   "version_type": "external"
>  }
> }'
{
  "took" : 12500,
  "timed_out" : false,
  "total" : 133724,
  "updated" : 0,
  "created" : 17996,
  "deleted" : 0,
  "batches" : 18,
  "version_conflicts" : 0,
  "noops" : 0,
  "retries" : {
    "bulk" : 0,
    "search" : 0
  },
  "throttled_millis" : 0,
  "requests_per_second" : -1.0,
  "throttled_until_millis" : 0,
  "failures" : [

So you see, "total" : 133724 and "created" : 17996.

Thank you!

2

Any ideas?

tnx

3

No ideas? Anyone?

4

Do you have any failures in the response?

5

failures are like the ones in this topic:

but as mentioned, i don't see any problems in the input data, shouldn't be any problems parsing...
maybe too much data?

6

This looks like a logstash failure. I'm more after having the entire response of the reindex call, which should include a failures array with useful information.

7

The whole response:

  "took" : 3472,
  "timed_out" : false,
  "total" : 136590,
  "updated" : 0,
  "created" : 0,
  "deleted" : 0,
  "batches" : 18,
  "version_conflicts" : 17996,
  "noops" : 0,
  "retries" : {
    "bulk" : 0,
    "search" : 0
  },
  "throttled_millis" : 0,
  "requests_per_second" : -1.0,
  "throttled_until_millis" : 0,
  "failures" : [
    {
      "index" : "mmsc-2017.06.05_new",
      "type" : "logs",
      "id" : "AVxdbFicS2Lv19EFCcix",
      "cause" : {
        "type" : "mapper_parsing_exception",
        "reason" : "failed to parse [expiry_time]",
        "caused_by" : {
          "type" : "illegal_argument_exception",
          "reason" : "Invalid format: \"38641620603:2057ADC5:\" is malformed at \"1620603:2057ADC5:\""
        }
      },
      "status" : 400
    },
    {
      "index" : "mmsc-2017.06.05_new",
      "type" : "logs",
      "id" : "AVxdlY6RS2Lv19EFDf_v",
      "cause" : {
        "type" : "mapper_parsing_exception",
        "reason" : "failed to parse [expiry_time]",
        "caused_by" : {
          "type" : "illegal_argument_exception",
          "reason" : "Invalid format: \"38631774539:153CFFFF:\" is malformed at \"1774539:153CFFFF:\""
        }
      },
      "status" : 400
    },
    {
      "index" : "mmsc-2017.06.05_new",
      "type" : "logs",
      "id" : "AVxcAbZ_S2Lv19EF9yV-",
      "cause" : {
        "type" : "mapper_parsing_exception",
        "reason" : "failed to parse [expiry_time]",
        "caused_by" : {
          "type" : "illegal_argument_exception",
          "reason" : "Invalid format: \"38631774539:141382D6:\" is malformed at \"1774539:141382D6:\""
        }
      },
      "status" : 400
    },
    {
      "index" : "mmsc-2017.06.05_new",
      "type" : "logs",
      "id" : "AVxb8VuGS2Lv19EF9vE4",
      "cause" : {
        "type" : "mapper_parsing_exception",
        "reason" : "failed to parse [expiry_time]",
        "caused_by" : {
          "type" : "illegal_argument_exception",
          "reason" : "Invalid format: \"38631774539:13C38316:\" is malformed at \"1774539:13C38316:\""
        }
      },
      "status" : 400
    }
  ]
}

Thanks

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.