we extended ECS for a specific use case where logstash is used to read some CSV, filter the data to match the custom ECS Extension and then send the data to elasticsearch. I am now wondering how to transfer my ECS extensions into an Index Template. Creating it manually in Kibana works for this single case but i plan on using the extension on more occasions. Is there some preexisting tool to create an Index Template from a custom ECS version or do i have to build my own tooling for this use case (leveraging the api to create the index template from the generated custom ecs mapping)?
In addition to the published ECS specification, the ECS GitHub repo also provides tooling for this type of use case. I'd suggest getting started here by looking through the ECS tooling usage guide. The guide provides instructions on downloading, installing dependencies, and getting started.
The --include option allows users to include their own custom schema definitions which are then merged with the official ECS fields. Once you run the generator with --include, ES index templates will be included in the generated artifacts:
The generated ES templates would be found in your ../myproject/out/generated/elasticsearch directory. These templates will include both the ECS fields and your custom fields definitions.
I already did the generation process but i am not quite sure if the only possibility to add the generated mapping to an index template/ index is doing it manually. The Beat modules create their own index templates and add the mappings themselves. Is the module which is doing that task available outside of beats?
Similar to Beats, the Elasticsearch output plugin in Logstash has an option available to specify your custom index template in the config.
The ECS tooling which generates custom mappings doesn't include built-in support for uploading them today. The generated mapping can be manually uploaded using the index template API (example) or through Index Management in Kibana.
The process linked to, based on the ECS scripts is ideal for fully custom indices.
When adding fields to Beats indices it's more tricky. The Beats do follow ECS, but they also define hundreds of additional custom fields that aren't in ECS.
So if you're adding fields on top of Beats modules, you'd have to look for guidance on this in the Beats docs. Note that I'm not sure there is documentation on how to do that for Beats. Every time I look, I come up empty. Perhaps you can ask about that in a Beats discuss post.
Component templates will certainly help solve for reusing settings, mappings, and aliases across multiple index templates if that's a need you require. The experimental Elasticsearch templates generated by the ECS tools today are still based on the legacy index template.
@jakobdoering for current Beats and modules the component templates won't help you: I think only the data sources in Elastic Agent are using component templates.
But I did find the documentation for adding fields to Beats templates, which you can use right now. The config option is setup.template.append_fields and is documented at
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.