Managed index templates, composition, and ECS

[Supermathie (Michael Brown)]

We are upgrading to Elastic 7 and at the same time attempting to move our entire pipeline to ECS. I'm wondering how template composition fits in with everything.

We primarily store transient logs and I'm wondering what the "best" way of managing index templates is. Right now we have templates for each index type and are specifying all (most) of the fields.

What we would like to do is specify only the things we're adding beyond the ECS spec.

It looks like the way we should be going about this is using the composable templates and specifying the unique fields and composing the ECS templates which we would to install ourselves from the ECS repo.

How are people installing their component templates? Right now we have all templates managed by logstash for consistency (and DO NOT want to do template installation manually).

One point of clarification I need arises from this documentation:

However, the Elasticsearch Index Templates it manages can be configured to be ECS-compatible by setting ecs_compatibility . By having an ECS-compatible template in place, we can ensure that Elasticsearch is prepared to create and index fields in a way that is compatible with ECS, and will correctly reject events with fields that conflict and cannot be coerced.

Does this only affect templates installed that we do not provide? e.g. if I specify the following template:

{
  "index_patterns": ["postgresql-*"],
  "settings": {
    "index": {
      "refresh_interval": "60s"
    }
  },
  "aliases": {
    "alllogs-{index}": {},
    "dblogs-{index}": {}
  },
  "mappings": {
    "properties": {
      "postgresql": {
        "cluster": { "type": "keyword" },
        "role":    { "type": "keyword" }
      }
    }
  }
}

via the elasticsearch output of logstash and also specify ecs_compatibility => "v1", will logstash ensure that my other fields are indexed according to ECS expectations?

[Supermathie (Michael Brown)]

As it turns out, this is a terrible thing to do since it appears that setting ecs_compatibility: v1 in the configuration causes logstash to completely ignore the template you specify to install.

So if you're using your own templates, don't set ecs_compatibility.

[jfs1 (Jean-François Stenuit)]

I'm in the same situation (upgrading old stack, migration to ECS). The way I intend to work is :

1. Integrate ECS into my own templates (done)
2. Edit my logstash filters to populate both legacy and ECS fields
3. Once stable, get rid of the legacy fields
4. Set ecs_compatibility when all logstash filters are updated and all back-end process are adapted to use ECS fields

I'm currently stuck in (2) as my logstash/elastic config doesn't seem to handle nested fields properly.

[Supermathie (Michael Brown)]

Where I am today:

I discovered that Elastic provides tooling to do EXACTLY what I want:

• specify our own custom fields in a meaningful way
• integrate the ECS definitions
• create the templates in a format that logstash can apply to elasticsearch

It took roughly a day to get tooled up and get something production-ready using the ECS tooling in that repository. I'd strongly suggest looking at that; you can specify your new fields (and aliases for backwards compatibility!) in YAML files and then generate your index templates from that.

So now we'll be doing nothing alluded to in the original post . But I do like this new method better.