# Dealing with dots in fields for the 2.0 upgrade
**URL:** https://discuss.elastic.co/t/dealing-with-dots-in-fields-for-the-2-0-upgrade/34562
**Category:** Elasticsearch
**Created:** [November 13, 2015, 8:57pm UTC](https://discuss.elastic.co/t/dealing-with-dots-in-fields-for-the-2-0-upgrade/34562 "2015-11-13T20:57:01Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@bstascavage](https://discuss.elastic.co/u/bstascavage)
#### Post date: [November 13, 2015, 8:57pm UTC](https://discuss.elastic.co/t/dealing-with-dots-in-fields-for-the-2-0-upgrade/34562/1 "2015-11-13T20:57:01Z")
When running the migration plugin, pretty much every index came back as failed, due to us having a template that automatically added an extra FIELD\_NAME.raw field that was no analyzed (See: [https://github.com/elastic/logstash/blob/v1.3.1/lib/logstash/outputs/elasticsearch/elasticsearch-template.json](https://github.com/elastic/logstash/blob/v1.3.1/lib/logstash/outputs/elasticsearch/elasticsearch-template.json)). I corrected this template for future indexes, but each of my existing indexes have offending fields. The only information I can really find is "reindex them", which seems like it is forcing me to do a lot of work to conform to the update. Regardless, I cannot find any information on how one would reindex while renaming fields to a new field, without pulling down every document, swapping out the field, and inserting it into a new index. I have almost 2k indexes and a few TBs of data, and while I could write a program to do this, there has to be a better way, right?
---
### Author:  [@shanec](https://discuss.elastic.co/u/shanec)
#### Post date: [November 14, 2015, 1:40am UTC](https://discuss.elastic.co/t/dealing-with-dots-in-fields-for-the-2-0-upgrade/34562/2 "2015-11-14T01:40:41Z")
Man I wish I could tell you this were easier.
First a moment of background. From a Logstash perspective, Aaron wrote a stickied post over [here](https://discuss.elastic.co/t/please-read-upgrading-logstash-and-elasticsearch-to-2-0/33791) about it. They added a ['de\_dot' filter](https://www.elastic.co/blog/introducing-the-de_dot-filter) to help with situations where you may not have control over the source fields. Dots in field names also caused problems in [Kibana](https://github.com/elastic/kibana/issues/3540). And they could cause really weird problems with field ambiguity, as seen [here](https://github.com/elastic/elasticsearch/issues/11337). Ugh... what a mess ever allowing them caused.
I know none of that helps your current situation, because the unfortunate answer is you're going to have to reindex. This is one of those breaking changes that just had to happen. We looked into ways to simplify the process to avoid a reindex or do it in place, but the options just weren't tenable unfortunately.
A few of the elasticsearch clients do have "reindex" functions built in. I've written [this script](https://github.com/eskibars/de-dot) which uses the perl reindex API, which you can see as an example.
---
### Author:  [@bstascavage](https://discuss.elastic.co/u/bstascavage)
#### Post date: [November 23, 2015, 10:25pm UTC](https://discuss.elastic.co/t/dealing-with-dots-in-fields-for-the-2-0-upgrade/34562/3 "2015-11-23T22:25:16Z")
I appreciate the script, but it does not cut it. It reindexes 1000 docs at a time; I have 5 billion documents.
This is going to sound harsh, and I know this going to be unpopular, but you guys dropped the ball with this. I am in this situation, not because I didn't think out my naming conventions ahead of time, but because the Logstash team introduced these fields as part of their template. I am in this situation because Elastic:
A. released an update that added these fields and
B. released another update that breaks on any data touched by A.
So why isn't Elastic offering a tool to migrate this data? Because from where I am standing, it seems to be that I am being told "Hey Brian, we caused you to be in a situation that prevents you from upgrading. Good luck fixing it on your own!" This is extremely unprofessional of you.
I have been spending cycles trying to reindex my data (again, 5 billion documents), and I guess I am dumb because there seems to be ZERO easy way to do it. At this point, why bother even using Elasticsearch if we are barred from ever upgrading in our current state?
Obviously I am upset, so again, sorry if this seems harsh. I just feel like I am spending my time doing something that Elastic should have been doing.
---
### Author:  [@shanec](https://discuss.elastic.co/u/shanec)
#### Post date: [November 24, 2015, 3:00am UTC](https://discuss.elastic.co/t/dealing-with-dots-in-fields-for-the-2-0-upgrade/34562/4 "2015-11-24T03:00:22Z")
I hear you on the pain, but any solution here would require a reindex. The best we can do is help you with tools that assist with the reindex. The script is one example and, if you're familiar with Logstash already, you can use the de-dot filter that was added there as an easy tool to assist with the process. See [https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06](https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06) for details of how to configure Logstash as a reindex tool.
In any case of what tool you decide to use, it sounds to me (correct me if I'm mishearing you) that your frustration is tied to the size of your index. That is, if you had a few million docs, either of the above two would chew through a thousand results at a time and be done pretty quickly, hardware permitting. It may be the case that the script or using the Logstash filter above may go very quick even on 5 billion documents. 1000 docs per batch may be fine, for example, if you get a throughput of 100 batches per second. But I'll leave the profiling aside because any case, there may be some things that may be done to help speed it up:
- Update your [translog fsync settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-translog.html) on the receiving cluster during the bulk reindex. ES 2.0 `fsync`s by default on every request, but you can disable that
- Use a different batch size: 1000 may not be optimal. This is not a 1-size-fits-all, as it varies by your hardware
- Run multiple instances of the reindexer of your choosing with the de-dot filter in parallel while providing each with a unique, non-overlapping query. This can be done in logstash by using the [query](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html) parameter. The same can be done with the script by adding a query body into the scroll. I've updated the script to show how to do this (line 49). Once you've split your data up a bit by query, you can then run several of them at the same time. Whether this helps or not will depend, again, on your hardware
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [November 24, 2015, 4:12pm UTC](https://discuss.elastic.co/t/dealing-with-dots-in-fields-for-the-2-0-upgrade/34562/5 "2015-11-24T16:12:53Z")
To add insult to injury, Elasticsearch works perfectly fine with dots in
field names: [https://github.com/elastic/elasticsearch/issues/14957](https://github.com/elastic/elasticsearch/issues/14957)
Dots were removed because they could cause ambiguity in the mapping. Better
to remove a feature since someone might shoot themselves in the foot. You
could run a patched version of Elasticsearch with the check in the mapper
service removed, and it will work perfectly fine.
---
### Author:  [@bstascavage](https://discuss.elastic.co/u/bstascavage)
#### Post date: [December 11, 2015, 8:02pm UTC](https://discuss.elastic.co/t/dealing-with-dots-in-fields-for-the-2-0-upgrade/34562/6 "2015-12-11T20:02:31Z")
So, after spending weeks reindexing to fix the issue introduced by the Logstash team, the migration tool finally clears me for upgrade. So I schedule a production window, take the cluster down, and...no, turns out the migration tool is a giant lie. _Closed_ indexes, which are not checked by the migration tool, _are_ checked by Elasticsearch 2.0.0 when it starts up! Therefore, without opening _every_ closed index, checking them, and closing them again, there is ZERO way to upgrade.
From where I am standing, and I know this is unpopular, but the 2.0.0 launch is straight-up embarrassing. There has been no regard for the end-user at all, and it is laughable that, have working for weeks to prep for the upgrade, I still am barred from upgrading! I am currently in a state where there is no resolution, and it is looking likely that there will be none. Further, it is frustrating that all of this 'incorrect' data was directly caused by Elastic code! I honestly can't think of a larger deployment mess.
My advice for your next upgrade: remember that you _do_ have existing users. For large-scale changes, use depreciation warnings, provide tools to actually migration data, and actually make sure your verification tools work. Don't leave your users out in the cold to fend for themselves, like this situation.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 11:31pm UTC](https://discuss.elastic.co/t/dealing-with-dots-in-fields-for-the-2-0-upgrade/34562/7 "2017-07-05T23:31:41Z")