Error when using template generated by ECS Tooling (1.7 and master) for datastream

Elastic Stack Elasticsearch
1

ECS Tooling (1.7.0 and master) generates template as following with my template-settings input:

{
   "index_patterns":[
      "my-data-stream*"
   ],
   "data_stream":{
   },
   "mappings":{
      "properties":{
         "message":{
            "type":"text"
         }
      }
   },
   "settings":{
      "index":{
         "lifecycle":{
            "name":"my-ilm-policy"
         },
         "mapping":{
            "total_fields":{
               "limit":10000
            }
         }
      }
   }
}

Pushing this template to elasticsearch (7.10.2), causing error:

"x_content_parse_exception","reason":"[4:3] [index_template] unknown field [mappings]"}.

It works with following format:

{
   "index_patterns":[
      "my-data-stream*"
   ],
   "data_stream":{
   },
   "template":{
      "mappings":{
         "properties":{
            "message":{
               "type":"text"
            }
         }
      }
   },
   "settings":{
      "index":{
         "lifecycle":{
            "name":"my-ilm-policy"
         },
         "mapping":{
            "total_fields":{
               "limit":10000
            }
         }
      }
   }
}

Note there's "template" here. I added "template" to template-settings for the tool without luck.

Is this supported by ECS Tooling? Any hidden options?
Thanks for any help,

Meimei

2

Currently, there are two different types of Elasticsearch index templates that the tooling generates:

  • The first is compatible with the legacy index template API, _template. This type will appear in the output directory at generated/elasticsearch/7/template.json for the 7.x compatible version.

  • The second is intended for use with the v2 index template API, _index_template. The generated example in the ECS tooling also makes use of component templates (more details here). The component template will be found at generated/elasticsearch/template.json with the component templates in the generated/elasticsearch/component directory.

Which template file are you trying to use in these examples?

Based on the error message and use of data_stream: { }, it looks like you're using the _index_template API?

You're correct that the exception from _index_template API is due to the aliases, mappings, and settings configs to be nested underneath template now:

POST _index_template/my-data-stream
{
  "index_patterns": [
    "my-data-stream*"
  ],
  "data_stream": {},
  "template": {
    "mappings": {
      "properties": {
        "message": {
          "type": "text"
        }
      }
    },
    "settings": {
      "index": {
        "lifecycle": {
          "name": "my-ilm-policy"
        },
        "mapping": {
          "total_fields": {
            "limit": 10000
          }
        }
      }
    }
  }
}

There's a known issue for the ECS tooling where custom settings for --template-settings aren't applied to the generated composable template. You may be experiencing the same problem if you're not seeing your --template-settings applied correctly. Work is in progress to fix this issue.

3

Eric, thanks for the response.
Yes, I am using _index_template API.
I was using generated/elasticsearch/7/template.json for the above API and ran into exception.
For now, we can manually edit either template.json files to have it work with _index_template (required by datastream).

Thanks for your help.

mm

4

@ebeahan is there option to generate one index template instead of component index in 1.8?
Thanks for your help,
Meimei

5

Yes, the legacy index templates are still generated as artifacts in the output: generated/elasticsearch/<version>/template.json.

6

Thanks.

We extend ECS schemas and named object starting with upper case, e.g., Link, Metric.Cpu.
The tool generates index template with composed_of as following:

"composed_of": [
    "ecs_2.0.0-dev_agent",
    "ecs_2.0.0-dev_base",
    "...",
    "ecs_2.0.0-dev_Metric.Cpu",
    "ecs_2.0.0-dev_Link" 
  ]

Elasticsearch throws exception due to upper case in the component_index names.
Is there an option to have all lower case for component index names?
Is there a way to have ONLY index template and NOT component templates in ECS Tooling 1.8 (just as legacy template)?

Thanks,

mm

7

This is a bug since a component template name must be lower case. I've created an issue to track here.

This isn't something the tooling generates today. It either the single template using the legacy template format or the current index template format using composable templates.

8

Thanks!

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.