Reindex with adding subdocuments (filebeat index 5.6 to 6.6 migration)

[Hubert_Lyczek (Hubert)]

Hello,

I'm testing upgrade from 5.6 to 6.6 currently. After upgrading ES, Kibana, Logstash and Beats I noticed that documents produced by beats are slightly different. That's ok, as all is as described in the documentation.

Example (old index):

{
  "host": "dev",
  "input_type": "log"
  ...
}

(new index):

{
  "host": {
    "name": "dev"
  },
  "input": {
    "type": "log"
  }
  ...
}

My problem:
I want to migrate old indices to a new structure. I will create new indexes anyway, as they will have a different mapping. I thought of using reindex functionality. Reindex API looks promising as it contains example how to rename fields during reindexing, but it allows adding new fields and modifying existing fields.

So something like this:

curl -i -H "Content-Type: application/json" --data '
{
  "source": {
    "index": "old"
  },
  "dest": {
    "index": "new",
    "type": "doc"
  },
  "script": {
    "source": "ctx._source.host.name = ctx._source.remove(\"host\"); ctx._source.index.type = ctx._source.remove(\"index_type\")"
  }
}' -X POST http://localhost:9200/_reindex

will produce errors. Problems are of course:

• host already exists and it is a String (do not have name field)
• index does not exist, and causes null_pointer_exception

I haven't find a good explanation of how to use context, what is contains, whether there is something like _destination.

Question:
Is it posible to achieve index reshape as described at the beginning?
if not, how to do this?

Thanks in advance.
Hubert

PS
This looks like a common problem, and this should be easy to solve. Am I missing something?

[Hubert_Lyczek (Hubert)]

This the solution to this problem:

POST _reindex

{        
      "source": {
        "index": "test_old"
      },                                                     
      "dest": {
        "index": "test_new",
        "type": "doc"
      },
      "script": {
        "source": "ctx._source.host = [\"name\": ctx._source.remove(\"host\")]; ctx._source.input = [\"type\": ctx._source.remove(\"input_type\")];"
      }

}

The main key is to use Painless Map syntax.
Now everything works fine.

Hubert