Automatic Index Creation

Elastic Stack Elasticsearch
61

@leandrojmp As you mentioned I am able to load template manually. But when trying to ingest below data through filebeat getting error as below.

sequence,component,tenant,service_id,session_id,timestamp,edr_version
2345671,31,INDAT,33,11711802425874023106770,1689041140000,V2

Please find the error as below.

\"reason\":\"data_stream [<sfw-{2023-07-11||/d{yyyy-MM-dd|UTC}}>] must not contain the following characters ['<','*','?','>','|',',','/','\\\\',' ','\\\"']\"}, dropping event!",

Thanks,
Debasis

62

What does your template looks like?

You need to share it, go at Index Management > Index Templates, select your template, click to edit and take a screenshot of the screen.

Also, on Dev Tools run this request and share the result:

GET _index_template/template-name

You probably checked the box Create data stream, which you should not check if you want daily indices, and there is something wrong in your index patterns configuration, it should be just sfx-*.

63

This time I created with Dev tools.

PUT _index_template/sfwidx_template
{
  "index_patterns": ["sfw-*"],
  "template": {
    "settings": {
      "number_of_shards": 4,
	  "number_of_replicas": 1
    },
    "mappings": {
      "_source": {
        "enabled": true
      },
      "properties": {
        "sequence": {
          "type": "keyword"
        },
        "component": {
          "type": "keyword"
        },
		"edr_version": {
          "type": "keyword"
		},
		"action_id": {
          "type": "integer"
		  },
		 "service_id": {
          "type": "integer"
		  },
		  "session_id": {
          "type": "keyword"
		  },
		  "tenant": {
          "type": "keyword"
		  },
		  "timestamp": {
          "type": "date"
		  }
      }
    },
    "aliases": {
      "sfwindex": { }
    }
  },
  "_meta": {
    "description": "sfw Index template"
  }
}

But when I do GET _index_template/sfwidx_template the O/P shows as below.

{
  "index_templates": [
    {
      "name": "sfwidx_template",
      "index_template": {
        "index_patterns": [
          "sfw-*"
        ],
        "template": {
          "settings": {
            "index": {
              "number_of_shards": "4",
              "number_of_replicas": "1"
            }
          },
          "mappings": {
            "_source": {
              "enabled": true
            },
            "properties": {
              "sequence": {
                "type": "keyword"
              },
              "component": {
                "type": "keyword"
              },
              "edr_version": {
                "type": "keyword"
              },
              "action_id": {
                "type": "integer"
              },
              "service_id": {
                "type": "integer"
              },
              "session_id": {
                "type": "keyword"
              },
              "tenant": {
                "type": "keyword"
              },
              "timestamp": {
                "type": "date"
              }
            }
          },
          "aliases": {
            "sfwindex": {}
          }
        },
        "composed_of": [],
        "_meta": {
          "description": "sfw Index template"
        }
      }
    }
  ]
}

Thanks,
Debasis

64

If you recreate the template, then the GET request will just show the template.

I asked this to see the template you had before, but since you recreated it, there is no need.

Have you test again to see if it is working now?

65

Thanks @leandrojmp able to perform in correct way. As you told data stream was chosen by default due to which getting error. Is there any way we can made data stream no while create template by using REST API command.
GET _index_template/template-name

Thanks,
Debasis

66

To create a data stream you just need to add this line in your index template.

"data_stream": {}

So, something like this:

PUT _index_template/template_name
{
  "index_patterns": ["data_stream_name"],
  "data_stream": {}
  "template": { the rest of your template }

But what is your end goal? In your original post you mentioned that you want to have daily indices, if you want to have daily indices you cannot use data streams, just normal indices.

With data streams you need to have an ILM policy because it uses an alias, the data stream name, and rollover for the backing indices.

You can read more about it here.

67

@leandrojmp I want to disable data stream in my Index template (how we can do while creating the template). By default it is getting enable while creating the template.

Thanks,
Debasis