We are upgrading to Elastic 7 and at the same time attempting to move our entire pipeline to ECS. I'm wondering how template composition fits in with everything.
We primarily store transient logs and I'm wondering what the "best" way of managing index templates is. Right now we have templates for each index type and are specifying all (most) of the fields.
What we would like to do is specify only the things we're adding beyond the ECS spec.
It looks like the way we should be going about this is using the composable templates and specifying the unique fields and composing the ECS templates which we would to install ourselves from the ECS repo.
How are people installing their component templates? Right now we have all templates managed by logstash for consistency (and DO NOT want to do template installation manually).
One point of clarification I need arises from this documentation:
However, the Elasticsearch Index Templates it manages can be configured to be ECS-compatible by setting
ecs_compatibility
. By having an ECS-compatible template in place, we can ensure that Elasticsearch is prepared to create and index fields in a way that is compatible with ECS, and will correctly reject events with fields that conflict and cannot be coerced.
Does this only affect templates installed that we do not provide? e.g. if I specify the following template:
{
"index_patterns": ["postgresql-*"],
"settings": {
"index": {
"refresh_interval": "60s"
}
},
"aliases": {
"alllogs-{index}": {},
"dblogs-{index}": {}
},
"mappings": {
"properties": {
"postgresql": {
"cluster": { "type": "keyword" },
"role": { "type": "keyword" }
}
}
}
}
via the elasticsearch output of logstash and also specify ecs_compatibility => "v1"
, will logstash ensure that my other fields are indexed according to ECS expectations?