Can't import index from ES1 to ES6

[dorinand]

Hello,

I am new with elastic search and I try to move indices from elasticsearch 1 to elasticsearch 6. There is way how to migrate from ES1 to ES2 and than ES2 to ES5 etc.. But it looks very complicated. I would like to try move all indexes from ES1 to new builded ES6.

I am able to download data and mapping via elasticdump tool. I was not able to upload data via this tool so I am trying to import data by creating new index and then import json file. When I create new index and import data using this command:

curl -H "Content-Type: application/json" -XPOST '[ipaddress]:9200/my_index/account/_bulk?pretty&refresh' --data-binary "@data.json"

there is response acknowledged : true, but only 28 of 1000 lines are imported. I assume that data are not consitent so I tried to set ignore malformed during creating of index by this command:

curl -XPUT '[ipaddress]:9200/my_index?pretty' -H 'Content-Type: application/json' -d '{"settings": {"index.mapping.ignore_malformed": true }}'

reply is successfull but when i try to import data, there is again only 28 lines.

Note: I tried above instructions with consistent data and everything works fine. Could anybody help me to solve my problem and answer my questions below?

My questions are:

1. Why I cant import all lines ?
2. How turn on ignore malformed ?
3. Is this good way to move to the newest ES?

Thank to everybody who will try to help me, have a nice day!

[abdon (Abdon Pijpelink)]

Do you see any errors in your Elasticsearch logs or as a response to your curl commands? Could you post those errors here?

Can you post a line of JSON that can be successfully ingested, and a line that cannot?

Answering your questions:

1. Without knowing anything about your data, or the elasticdump tool , my guess is that your data has multiple document types in the same index. This was allowed in version 1, but no longer in version 6. You will have to either split up your data into separate indexes (one index per type), or normalize all document types to a single type.

2. I don't see how ignore_malformed would help.

3. Have you looked at the _reindex API? You can use it to get data from indexes in a remote cluster into a local index. You could consider using that to get all data from your old v1 cluster into your new v6 cluster. Check out the docs here: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html#reindex-from-remote

[dorinand]

Hello, thank you for your reply.

1.I solved my issue, there was problem with format of some lines (missing parenthesis). I repaired it and it works.
2. I did not understand ignore_malformed properly, you are right it would not help. My mistake.

3. I will try _reindex API, because there is big indexes (tens of GB) It looks like great way to move big indices.

Thank you for your advice.

[dorinand]

Hi Abdon,

it's hard for me to share data with you, it is internal data of company.

I tried _reindex API. It update few thousand docs but I received a lot of errors, e.g.:

  "index" : [my_index],
  "type" : "mp_media_processor",
  "id" : [my_id],
  "cause" : {
    "type" : "mapper_parsing_exception",
    "reason" : "failed to parse [level]",
    "caused_by" : {
      "type" : "number_format_exception",
      "reason" : "For input string: \"INFO\""
    }
  },
  "status" : 400
},

I compare banned doc with doc which passed. The difference is that doc that passed have "level" is an integer 200 ("level":200) and doc that cannot, have "level" set to INFO ("level":"INFO"). So if I understand correctly, there is different document type so I cant move whole index from ES1 to ES6 nad I have to split it to more indexes?

There is more erorrs like: "illegal_argument_exception", "Limit of total fields [1000] in index [my_index] has been exceeded", "Rejecting mapping update to [my_index] as the final mapping would have more than 1 type: [index-api, index-microservice]", "failed to parse", and few more. Can you help me to resolve this issue? Do you need more docs example? I understant that I can set total field lmiit to bigger number than 1000, but what about ather errors? Thank you for your help.

[abdon (Abdon Pijpelink)]

Did you define an explicit mapping for your target index? If you don't define an explicit mapping for the index, Elasticsearch will use dynamic mapping, and it will "guess" what the types of your fields are. That seems to be the issue with the level field: it was dynamically mapped to a long, so it won't accept any documents where that field is a string like "INFO".

A 1000 fields is a lot for a single index, that's why there's a soft limit of a 1000 fields. You can increase this limit though, with the index.mapping.total_fields.limit setting. You can apply this setting when you create an index, or apply it afterwards.