ES 6.5.0 : Custom mapping isn't taken into account whereas it is in 6.4.2

[mbutton77 (Maxence Button)]

I've lost some time trying to figure out what was wrong with my custom mapping while working with the brand new ElasticStack 6.5.0.
Just out of curiosity I tried my configuration on a 6.4.2 and everything went fine.

Here's my logstash config file :

input { 
    file {
        path => "stuff.csv"
        start_position => "beginning"
        sincedb_path => "/dev/null"
    }    
}

filter {
    csv {
        skip_header => "true"
        columns => ["id", "name", "latitude", "longitude"]
    }
    mutate {
        rename => {
            "latitude"  => "[location][lat]"
            "longitude" => "[location][lon]"
        }
        convert => {
            "[location][lat]" => "float"
            "[location][lon]" => "float"
        }
        remove_field => [
            "host",
            "@version",
            "message",
            "path"
        ]
    }
}

output {
    elasticsearch { 
        template => "stuff_mapping.json"
        index => "logstash-stuff"
        hosts => ["localhost:9200"] 
    }
    stdout { codec => rubydebug }
}

And here is my custom mapping (template) :

{
  "template": "logstash-stuff",
  "version": 1,
  "settings": {
    "index.refresh_interval": "5s"
  },
  "mappings": {
    "_default_": {
      "properties": {
        "id": {
          "type": "text",
          "norms": false,
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "location": {
          "type": "geo_point"
        },
        "name": {
          "type": "text",
          "norms": false,
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    }
  }
}

When I open the elastic mapping in Kibana or via cURL againt ES 6.5.0, the mapping remain unchanged (as if nothing was passed, whereas I can see Logstash indeed passing the custom mapping).

But against ES 6.4.2, everything's fine : my location has been correctly indexed as a geo_point.

I checked the release note, nothing seems to indicate that I should have changed a thing.

Can somebody please tell me what I did wrong ?

Thanks.

[mbutton77 (Maxence Button)]

Damn, I found the problem ...
Nothing wrong with ES 6.5.0, just a trick you've got to have in mind when applying custom mappings.

When I first created my custom mapping, something was wrong with it, thus ES decided to not apply it and to determine the default mapping from what it could read.

And my mistake came from the fact that, by erasing everything thanks to that command :

curl -XDELETE localhost:9200/_all

I thought that everything was wiped out and almost brand new.
But that's not the way it is : I have the impression that a reference to the first custom mapping is kept somewhere and you have no way to delete it.

Unless you precise the magical option :

template_overwrite => "true"

And that's why when I switched to the 6.4.2, it worked at once : the mapping (which was correct this time) had never been applied and ES did take it into account.

To be sure of that, I switched back to ES 6.5.0 with the magical option, and it worked as it should have been.

So no problem with ES 6.5.0, just a silly mistake induced by a weird behavior.