Index ignoring template

[turnlikeawheel]

Disclaimer: I'm a bit new to ES/Filebeat so I may be misunderstanding something.

The problem is that ES is indexing the Filebeat field "beat.hostname" as text rather than keyword on seemingly random days when the daily index is created. The problem I'm having with this is that I have some visualizations that aggregate this field. When I load them now, I'm getting the "Courier Fetch: x of x shards failed" error because it can't aggregate the field from those indices.

The template for Filebeat 6.1.0 seems to be correct and 95% of my indices are fine.

The relevant template sections:

"filebeat-6.1.0": {
"order": 1,
"index_patterns": [
"filebeat-6.1.0-"
],
"settings": {
"index": {
"mapping": {
"total_fields": {
"limit": "10000"
}
},
"refresh_interval": "5s",
"number_of_routing_shards": "30",
"number_of_shards": "2"
}
},
"mappings": {
"doc": {
"_meta": {
"version": "6.1.0"
},
"date_detection": false,
"dynamic_templates": [
{
"fields": {
"mapping": {
"type": "keyword"
},
"match_mapping_type": "string",
"path_match": "fields."
}
},

.....

     "beat": {
        "properties": {
          "version": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "name": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "hostname": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "timezone": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },

But the mapping for that day shows:

"filebeat-6.1.0-2017.12.16": {
"mappings": {
"doc": {
"properties": {
"@timestamp": {
"type": "date"
},
"beat": {
"properties": {
"hostname": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}

I'm pretty sure I'm missing something, but if someone could point me in the right direction, that would be fantastic.

Thanks!

[luiz.santos]

Hi @turnlikeawheel,

It happens that elasticsearch creates by default the mapping for type text and keyword for the same field.

{
	"beat": {
		"properties": {
			"hostname": {
				"type": "text",
				"fields": {
					"keyword": {
						"type": "keyword",
						"ignore_above": 256
					}
				}
			}
		}
	}
}

To use the keyword field you should do "hostname.keyword".

Hope it helps.

Cheers,
LG