How does custom template work in elasticsearch?

Elastic Stack Elasticsearch
1

I've asked a question a while back, but cant seem to find a proper solution or a work around...

I currently have a filter that parse a certain amount of logs and aggregate it all together to finalize a final log (each A, B, C, etc part logs have an identified called orderId) and we query for orderId as a keyword to aggregate proper ones (otherwise, the query would return incorrect results).

I have a custom template here:

{
	"order" : 1,
	"template": "logstash-transaction-*",
	"settings": {
		"index.refresh_interval": "5s"
	},
	"mappings": {
		"logs": {
			"_all": {
				"enabled": true,
				"omit_norms": true
			},
			"dynamic_templates": [
				{
					"message_field": {
						"match": "message",
						"match_mapping_type": "string",
						"mapping": {
							"type": "string",
							"index": "analyzed",
							"omit_norms": true
						}
					}
				},
				{
					"string_fields": {
						"match": "*",
						"match_mapping_type": "string",
						"mapping": {
							"type": "string",
							"index": "not_analyzed",
							"ignore_above": 256
						}
					}
				}
			],
			"properties": {
				"orderId": {
					"type": "keyword"
				}
			}
		}
	}
}

However, for some reason, once our index would be created after delete and clean restart, it would not correctly map the template... I suspect its due to this template for some reason that exist that I've asked about a long time ago here (Forcing only a particular template on index?). To summarize, I suspected that the index was affected by the other template that presumably was generated by logstash because when I do GET _template I see there are two different templates that could be applied to my index of logstash-transaction-%{+YYYY.MM.dd}

I thought this (Disable logstash default template creation) would solve it, but it stops the logstash-* template from generating, but my custom template is still not applied...

This is my output-elasticsearch.conf, could anyone care to expand on this behavior? Am I missing something here??

output {
    if (![log]) {
        elasticsearch {
            hosts => [ "${OUTPUT_ELASTICSEARCH_HOSTS}" ]
            index => "${OUTPUT_ELASTICSEARCH_INDEX}"
            action => "${OUTPUT_ELASTICSEARCH_ACTION:index}"
            document_id => "%{logGUID}"
            document_type => "${OUTPUT_ELASTICSEARCH_DOCUMENT_TYPE}"
            retry_on_conflict => 50
            template_name => "logstash-transaction"
            manage_template => true
            template_overwrite => true
            template => "${CONFIG_DIR}/_/logs-elasticsearch-template.json"
         }
    }
}
2

What version are you on?

This needs to match your index, so I'd make sure of that. Also;

That needs to match your mapping type, so check it does as well.

3

I'm using Elasticsearch 5.6.8

As for document_type, I'm assuming thats the mappings's name?

4

document_type needs to be the same as logs.

5

Ah ok, that's what I did then. I think that might have fixed it, seems like the environmental variable did not exist so it used the default template.

Now it seems like it makes every string field a keyword type, is there a way to only make it keyword for specifically orderId? I guess it has to deal with the mapping under string_fields vs message_fields.
like this?

"string_fields": {
	"match": "*",
	"match_mapping_type": "string",
	"mapping": {
		"type": "string",
		"index": "analyzed",
		"ignore_above": 256
	}
}
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.