# Managed index templates, composition, and ECS

**URL:** <https://discuss.elastic.co/t/managed-index-templates-composition-and-ecs/266734>\
**Category:** Logstash\
**Tags:** ecs-elastic-common-schema\
**Created:** [March 9, 2021, 6:10pm UTC](https://discuss.elastic.co/t/managed-index-templates-composition-and-ecs/266734 "2021-03-09T18:10:09Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Supermathie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/supermathie/32/44936_2.png) [@Supermathie](https://discuss.elastic.co/u/Supermathie)\
**Post date:** [March 9, 2021, 6:10pm UTC](https://discuss.elastic.co/t/managed-index-templates-composition-and-ecs/266734/1 "2021-03-09T18:10:09Z")

</div>

We are upgrading to Elastic 7 and at the same time attempting to move our entire pipeline to ECS. I'm wondering how template composition fits in with everything.

We primarily store transient logs and I'm wondering what the "best" way of managing index templates is. Right now we have templates for each index type and are specifying all (most) of the fields.

What we would _like_ to do is specify only the things we're adding beyond the ECS spec.

It looks like the way we should be going about this is using the [composable templates](https://www.elastic.co/guide/en/elasticsearch/reference/current//index-templates.html) and specifying the unique fields and composing the ECS templates which we would to install ourselves [from the ECS repo](https://github.com/elastic/ecs/tree/1.8/generated/elasticsearch/component).

How are people installing their component templates? Right now we have all templates managed by logstash for consistency (and DO NOT want to do template installation manually).

One point of clarification I need arises from [this documentation](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#_compatibility_with_the_elastic_common_schema_ecs):

> However, the Elasticsearch Index Templates it manages can be configured to be ECS-compatible by setting [`ecs_compatibility`](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-ecs_compatibility). By having an ECS-compatible template in place, we can ensure that Elasticsearch is prepared to create and index fields in a way that is compatible with ECS, and will correctly reject events with fields that conflict and cannot be coerced.

Does this only affect templates installed that we _do not_ provide? e.g. if I specify the following template:

```json
{
  "index_patterns": ["postgresql-*"],
  "settings": {
    "index": {
      "refresh_interval": "60s"
    }
  },
  "aliases": {
    "alllogs-{index}": {},
    "dblogs-{index}": {}
  },
  "mappings": {
    "properties": {
      "postgresql": {
        "cluster": { "type": "keyword" },
        "role": { "type": "keyword" }
      }
    }
  }
}

```

via the elasticsearch output of logstash and also specify `ecs_compatibility => "v1"`, will logstash ensure that my _other_ fields are indexed according to ECS expectations?

---

<div class="post-metadata">

**Author:** ![Supermathie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/supermathie/32/44936_2.png) [@Supermathie](https://discuss.elastic.co/u/Supermathie)\
**Post date:** [March 12, 2021, 12:54am UTC](https://discuss.elastic.co/t/managed-index-templates-composition-and-ecs/266734/2 "2021-03-12T00:54:44Z")

</div>

As it turns out, this is a terrible thing to do since it appears that setting `ecs_compatibility: v1` in the configuration causes logstash to completely ignore the template you specify to install.

So if you're using your own templates, don't set `ecs_compatibility`.

---

<div class="post-metadata">

**Author:** ![jfs1](https://avatars.discourse-cdn.com/v4/letter/j/439d5e/32.png) [@jfs1](https://discuss.elastic.co/u/jfs1)\
**Post date:** [March 15, 2021, 9:56am UTC](https://discuss.elastic.co/t/managed-index-templates-composition-and-ecs/266734/3 "2021-03-15T09:56:05Z")

</div>

I'm in the same situation (upgrading old stack, migration to ECS). The way I intend to work is :

1. Integrate ECS into my own templates (done)
2. Edit my logstash filters to populate both legacy and ECS fields
3. Once stable, get rid of the legacy fields
4. Set `ecs_compatibility` when all logstash filters are updated and all back-end process are adapted to use ECS fields

I'm currently stuck in (2) as my logstash/elastic config doesn't seem to handle nested fields properly.

---

<div class="post-metadata">

**Author:** ![Supermathie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/supermathie/32/44936_2.png) [@Supermathie](https://discuss.elastic.co/u/Supermathie)\
**Post date:** [March 19, 2021, 8:04pm UTC](https://discuss.elastic.co/t/managed-index-templates-composition-and-ecs/266734/4 "2021-03-19T20:04:40Z")

</div>

Where I am today:

I discovered that Elastic provides [tooling](https://github.com/elastic/ecs/) to do EXACTLY what I want:

- specify our own custom fields in a meaningful way
- integrate the ECS definitions
- create the templates in a format that logstash can apply to elasticsearch

It took roughly a day to get tooled up and get something production-ready using the ECS tooling in that repository. I'd strongly suggest looking at that; you can specify your new fields (and aliases for backwards compatibility!) in YAML files and then generate your index templates from that.

So now we'll be doing _nothing_ alluded to in the original post 😆. But I do like this new method better.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 16, 2021, 8:05pm UTC](https://discuss.elastic.co/t/managed-index-templates-composition-and-ecs/266734/5 "2021-04-16T20:05:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.