Elastic Common Schema generator scripts

[Nybble (Sebastien Lehuede)]

Hello,

I want to generate an ES mapping JSON file from my $fields.yml files with scripts on ECS Github (https://github.com/elastic/ecs/tree/master/scripts) but I can't figure out how to use these python scripts.

Is it possible to get some help on how to use generator scripts ?

I assume it's possible to use them as mention here : https://github.com/elastic/ecs/blob/master/CONTRIBUTING.md

Thanks !
Sébastien.

[webmat (Mathieu Martin)]

Bonjour Sébastien,

The ECS scripts are first and foremost made to generate the ECS documentation and other resources.

You're right that it can be used to generate Elasticsearch templates as well, though. We use it to generate the sample templates provided in the Github repo.

Many users have been using the repo's script to generate their templates, and until very recently, the process was quite painful. A recent pull request has added the ability to include a second directory of YAML files that would contain a user's custom fields. There's a few more improvements we want to do, to make this process smoother.

Since you haven't specifically asked about the dev environment, I assume you're already set up with Python 2.7 and a recent enough Golang. If you need help with this as well, let me know.

Once you're up & running with the dev environment, you can create additional schema files in another directory outside of your clone of the ECS repo (you'll likely want to track your own changes on those). You can figure out how to create these YAML files by looking at the ones that form ECS, in schemas/, as well as checking out the readme in the schemas directory.

Once you have one or more schema files in your directory, you can generate all files based on ECS + your fields in a few ways:

INCLUDE=/path/to/your/schemas make generate
or
python scripts/generator.py --include /path/to/your/schemas

This will re-generate the Elasticsearch templates (and a bunch of other files) in place, directly in your clone the ECS repo. I know this isn't ideal. But as noted in the issue linked above, we want to add other niceties to this process, like specifying an output directory, selecting which files to generate instead of generating everything, and so on.

Until then, this is what we have Let me know if this helps. And if inspiration strikes while you work on this, pull requests are welcome!

Mathieu

[Nybble (Sebastien Lehuede)]

Hi Mathieu,

This is exactly what I needed ! Many thanks
Just Golang was missing on my machine but now all fine.

I'm able to generate Elasticsearch template, included my fields successfully and it's not that annoying.

Maybe I will work on a little script who consume the CSV where my fields are defined and documented to put the non-existing fields in schema files in right format.

Thanks !
Sébastien.