Because my project is mainly for aggregation (not for searching), I want to make all the string type fields by default not _analyzed. I used to use Elasticsearch from 1.7 to 2.2, they all support user defined dynamic_template, but when I switch to Elasticsearch 2.3, according to my test, it does not support user defined dynamic_template.
This is my logstash .conf file, I used this .conf file for two tests, one is using dynamic_template, another is using root level properties definition to define string fields to be not analysed.
input{stdin{}}
filter {
json { source => "message" }
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => "staging-elkstack:9200"
index => "template_test"
template_name => "elkstats_template"
}
}
sample input
{ "longitude":-73.758, "latitude":41.0291}
Test 1:
Use root level properties definition to define host and message, they are not_analyzed as I expected
curl -XPUT 'http://staging-elkstack:9200/_template/elkstats_template' -d '
{
"order" : 3,
"template": "template_test*",
"mappings" : {
"log" : {
"_all" : {"enabled" : false, "omit_norms" : true},
"properties" : {
"@timestamp": { "type": "date", "doc_values" : true },
"@version": { "type": "string", "index": "not_analyzed", "doc_values" : true },
"message": { "type": "string", "index": "not_analyzed", "doc_values" : true },
"host": { "type": "string", "index": "not_analyzed", "doc_values" : true }
}
}
}
}
'
As you can see in the kibana settings interface, now the message, and host fields are not_analyzed
Test 2:
I used this script to define the dynamic_template for strings to be not_analyzed in general
curl -XPUT 'http://staging-elkstack:9200/_template/elkstats_template' -d '
{
"order" : 3,
"template": "template_test*",
"mappings" : {
"log" : {
"_all" : {"enabled" : false, "omit_norms" : true},
"dynamic_templates" : [ {
"message_field" : {
"match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "not_analyzed", "omit_norms" : true
}
}
}, {
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "not_analyzed", "omit_norms" : true
}
}
}
],
"properties" : {
"@timestamp": { "type": "date", "doc_values" : true },
"@version": { "type": "string", "index": "not_analyzed", "doc_values" : true }
}
}
}
}
'
As you can see, the host and message fields are still analyzed (which means that the definition in "dynamic_template" section is not used here)
I really need the dynamic_template settings to ensure string type fields are not_analyzed, what can I do to make it work?