Reindexing, mapping and templating errors

[Celso_Santos]

Hello everyone,

I've recently upgraded from ES 5.5 to ES 6.2 by snapshotting the old indexes and restoring them onto 6.2. So far, so good.

Now, I've started to notice that, because there was still some filebeat 5.5.x I upgraded it to 5.6.8 and then to 6.2.2. Again, so far so good.

Thing is, my indexes started to complain about conflicts in a couple of fields, particularly with geoip.postal_code and geoip.coordinates, which is causing issues on the imported visualizations, specifically the map view.

I've already fixed my template file to make it compatible with 6.2 by defining my template as the following:

PUT {{protocol}}://{{host}}:{{port}}/_template/proxy HTTP/1.1
content-type: {{contentType}}
Authorization: {{authorization}}

{
  "index_patterns": ["proxy-*"],
  "settings": {
    "index.refresh_interval": "5s",
    "number_of_shards" : "2",
    "number_of_replicas" : "2"
  },
  "mappings": {
    "doc": {
      "dynamic_templates": [
        {
          "template1": {
            "mapping": {
              "doc_values": true,
              "ignore_above": 1024,
              "index": false,
              "type": "{dynamic_type}"
            },
            "match": "*"
          }
        }
      ],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "message": {
          "type": "text",
          "index": true
          "doc_values": false
        },
        "offset": {
          "type": "long",
          "doc_values": true
        },
        "coordinates"  : {
          "type" : "geo_point"
        }
      }
    }
  }
}

I also had a bunch of proxy-yyyy-mm-dd indexes which I reindexed into a single one before applying said template, named proxy-2017.
After creating the template I created a new index called proxy-2017-fixed onto which I'm trying to reindex my data. I've already fixed a bunch of errors (such as "mapping would have more than 1 type: [log, doc]")

Now, I want to reindex the proxy-2017 data into proxy-2017-fixed but I'm always getting an error that "[text] fields do not support doc values". What gives? I've already modified my template a bunch of times to no avail. What am I doing wrong? How can I fix this?

This is my reindex call:

POST {{protocol}}://{{host}}:{{port}}/_reindex HTTP/1.1
content-type: {{contentType}}
Authorization: {{authorization}}

{
  "source": {
    "index": "proxy-2017"
  },
  "dest": {
    "index": "proxy-2017-fixed"
  },
  "script": {
    "source": " ctx._type = 'doc'; "
  }
}

I was under the impression that this process would be much simpler than it is proving to be.

Best Regards.

[abdon (Abdon Pijpelink)]

The problem is with the dynamic template in your index template: defining "doc_values": true for all types causes issues, because doc values are not supported for text type fields. Whenever you will index a string that has not been explicitly mapped, this will lead to an error, as you saw when trying to reindex your documents.

Now, "doc_values": true is already the default for those field types that do support doc values. So, all you need to do is remove the "doc_values": true from your dynamic template and all should work fine.