Mapper for [clientip] conflicts with existing mapping in other types:\n[mapper [clientip] has different [norms] values, cannot change from disable to enabled]

[chrisan]

Elasticsearch 6.2.4

We use logstash+elasticsearch to get some bandwidth metrics on our cloudfront usage. This is how logstash parses things: https://gist.github.com/chrisan/1c5ce5beacfc0e124d39fa842f051857#file-logstash-api-conf

This generates indices such as: https://gist.github.com/chrisan/1c5ce5beacfc0e124d39fa842f051857#file-indicies

With mappings like: https://gist.github.com/chrisan/1c5ce5beacfc0e124d39fa842f051857#file-mappings

I was asked to get distinct IP addresses and I tried using an aggregate query:

{
  "size": 0,
    "aggs" : {
        "distinct_ips" : {
            "filter" : { "term": { "company" : "XXX" } },
            "aggs" : {
                "cardinality" : { "cardinality": {"field": "clientip"	} }
            }
        }
    }
}

But this returns:

{
    "error": {
        "root_cause": [
            {
                "type": "illegal_argument_exception",
                "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [clientip] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
            }
        ],
        "type": "search_phase_execution_exception",
        "reason": "all shards failed",
        "phase": "query",
        "grouped": true,
        "failed_shards": [
            {
                "shard": 0,
                "index": "logstash-2018.01.01",
                "node": "dO1JCnAnSmmk5EfDmfYgqQ",
                "reason": {
                    "type": "illegal_argument_exception",
                    "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [clientip] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
                }
            }
        ]
    },
    "status": 400
}

So I tried to update that with

PUT /*/_mapping/_doc?update_all_types

{
  "properties": {
    "clientip": {
      "type":     "text",
      "fielddata": true
    }
  }
}

Which returns:

{
    "error": {
        "root_cause": [
            {
                "type": "illegal_argument_exception",
                "reason": "Mapper for [clientip] conflicts with existing mapping in other types:\n[mapper [clientip] has different [norms] values, cannot change from disable to enabled]"
            }
        ],
        "type": "illegal_argument_exception",
        "reason": "Mapper for [clientip] conflicts with existing mapping in other types:\n[mapper [clientip] has different [norms] values, cannot change from disable to enabled]"
    },
    "status": 400
}

What am I doing wrong?

[chrisan]

Figured it out, had to delete everything and then update the logstash template with

{
		"template": "logstash",
		"order": 0,
		"version": 60001,
		"index_patterns": [
			"logstash-*"
		],
		"settings": {
			"index": {
				"refresh_interval": "5s"
			}
		},
		"mappings": {
			"_default_": {
				"dynamic_templates": [
					{
						"message_field": {
							"path_match": "message",
							"match_mapping_type": "string",
							"mapping": {
								"type": "text",
								"norms": false
							}
						}
					},
					{
						"string_fields": {
							"match": "*",
							"match_mapping_type": "string",
							"mapping": {
								"type": "text",
								"norms": false,
								"fields": {
									"keyword": {
										"type": "keyword",
										"ignore_above": 256
									}
								}
							}
						}
					}
				],
				"properties": {
					"@timestamp": {
						"type": "date"
					},
					"@version": {
						"type": "keyword"
					},
					"clientip": {
						"type":     "text",
						"fields": {
						"keyword": { 
						  "type": "keyword"
						}
					  }
					},
					"company": { 
					  "type": "text",
					  "fields": {
						"keyword": { 
						  "type": "keyword"
						}
					  }
					},
					"geoip": {
						"dynamic": true,
						"properties": {
							"ip": {
								"type": "ip"
							},
							"location": {
								"type": "geo_point"
							},
							"latitude": {
								"type": "half_float"
							},
							"longitude": {
								"type": "half_float"
							}
						}
					}
				}
			}
		},
		"aliases": {}
	}

And then I could query with

"aggs": {
   "distinct_ips": {
     "cardinality": {
       "field": "clientip.keyword"
     }
   }
 }