Migrating fields with dots for ES 2.0 upgrade

Elastic Stack Elasticsearch
1

Is there a way to migrate a bunch of fields in our indices from names that have dots in them (e.g., "http.responseCode") to nested objects?

I'd like to upgrade our ES 1.6 cluster to ES 2.0, but in letting the Migration Checker plugin do its thing, it turns out we have an issue with dots in the field names in nearly all our indices. (It looks like one of our Java logging libraries happily included dots in the field names of data it sends to Logstash and thus through to ES, rather than nesting the data structure — so we have field names like "http.responseCode" and "db.queryTime".)

Is there some way to easily fix this in our indices? I've hunted and hunted for pre-existing solutions, to no avail — and dots in field names is one of the issues that absolutely prevents an ES 2.0 upgrade.

Thanks in advance.

2

You need to reindex to fix this, here's how I do it - https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06

3

Thanks — but does that natively just handle re-indexing the fields with dots in their names to nested data structures? From what I can read, it instead will just error out on the attempt to re-insert the documents into the ES 2.0 cluster, since the fields have dots in their names...

4

It's just an example of a process to make the reindex process easier. You do need to massage the data to remove the dots.