# Automatic Index Creation

**URL:** <https://discuss.elastic.co/t/automatic-index-creation/361788>\
**Category:** Elasticsearch\
**Created:** [June 20, 2024, 11:06am UTC](https://discuss.elastic.co/t/automatic-index-creation/361788 "2024-06-20T11:06:23Z")\
**Posts on this page:** 7\
**Page:** 4

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [July 23, 2024, 11:10am UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/61 "2024-07-23T11:10:46Z")

</div>

@leandrojmp As you mentioned I am able to load template manually. But when trying to ingest below data through filebeat getting error as below.

```auto
sequence,component,tenant,service_id,session_id,timestamp,edr_version
2345671,31,INDAT,33,11711802425874023106770,1689041140000,V2

```

Please find the error as below.

`\"reason\":\"data_stream [<sfw-{2023-07-11||/d{yyyy-MM-dd|UTC}}>] must not contain the following characters ['<','*','?','>','|',',','/','\\\\',' ','\\\"']\"}, dropping event!",`

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 23, 2024, 12:16pm UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/62 "2024-07-23T12:16:35Z")

</div>

What does your template looks like?

You need to share it, go at Index Management \> Index Templates, select your template, click to edit and take a screenshot of the screen.

Also, on Dev Tools run this request and share the result:

```auto
GET _index_template/template-name

```

You probably checked the box _Create data stream_, which you should not check if you want daily indices, and there is something wrong in your index patterns configuration, it should be just `sfx-*`.

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [July 23, 2024, 12:32pm UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/63 "2024-07-23T12:32:15Z")

</div>

> [@leandrojmp](#):
>
> What does your template looks like?

This time I created with Dev tools.

```auto
PUT _index_template/sfwidx_template
{
  "index_patterns": ["sfw-*"],
  "template": {
    "settings": {
      "number_of_shards": 4,
	  "number_of_replicas": 1
    },
    "mappings": {
      "_source": {
        "enabled": true
      },
      "properties": {
        "sequence": {
          "type": "keyword"
        },
        "component": {
          "type": "keyword"
        },
		"edr_version": {
          "type": "keyword"
		},
		"action_id": {
          "type": "integer"
		  },
		 "service_id": {
          "type": "integer"
		  },
		  "session_id": {
          "type": "keyword"
		  },
		  "tenant": {
          "type": "keyword"
		  },
		  "timestamp": {
          "type": "date"
		  }
      }
    },
    "aliases": {
      "sfwindex": { }
    }
  },
  "_meta": {
    "description": "sfw Index template"
  }
}

```

But when I do `GET _index_template/sfwidx_template` the O/P shows as below.

```auto
{
  "index_templates": [
    {
      "name": "sfwidx_template",
      "index_template": {
        "index_patterns": [
          "sfw-*"
        ],
        "template": {
          "settings": {
            "index": {
              "number_of_shards": "4",
              "number_of_replicas": "1"
            }
          },
          "mappings": {
            "_source": {
              "enabled": true
            },
            "properties": {
              "sequence": {
                "type": "keyword"
              },
              "component": {
                "type": "keyword"
              },
              "edr_version": {
                "type": "keyword"
              },
              "action_id": {
                "type": "integer"
              },
              "service_id": {
                "type": "integer"
              },
              "session_id": {
                "type": "keyword"
              },
              "tenant": {
                "type": "keyword"
              },
              "timestamp": {
                "type": "date"
              }
            }
          },
          "aliases": {
            "sfwindex": {}
          }
        },
        "composed_of": [],
        "_meta": {
          "description": "sfw Index template"
        }
      }
    }
  ]
}

```

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 23, 2024, 12:45pm UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/64 "2024-07-23T12:45:54Z")

</div>

If you recreate the template, then the `GET` request will just show the template.

I asked this to see the template you had before, but since you recreated it, there is no need.

Have you test again to see if it is working now?

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [July 24, 2024, 7:13am UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/65 "2024-07-24T07:13:04Z")

</div>

> [@leandrojmp](#):
>
> Have you test again to see if it is working now?

Thanks @leandrojmp able to perform in correct way. As you told _data stream_ was chosen by default due to which getting error. Is there any way we can made data stream no while create template by using REST API command.  
`GET _index_template/template-name`

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 24, 2024, 12:27pm UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/66 "2024-07-24T12:27:50Z")

</div>

> [@Debasis\_Mallick](#):
>
> Is there any way we can made data stream no while create template by using REST API command.

To create a data stream you just need to add this line in your index template.

```auto
"data_stream": {}

```

So, something like this:

```auto
PUT _index_template/template_name
{
  "index_patterns": ["data_stream_name"],
  "data_stream": {}
  "template": { the rest of your template }

```

But what is your end goal? In your original post you mentioned that you want to have daily indices, if you want to have daily indices you cannot use data streams, just normal indices.

With data streams you need to have an ILM policy because it uses an alias, the data stream name, and rollover for the backing indices.

You can read more about it [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html).

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [July 24, 2024, 12:47pm UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/67 "2024-07-24T12:47:44Z")

</div>

> [@leandrojmp](#):
>
> To create a data stream you just need to add this line in your index template.

@leandrojmp I want to disable data stream in my Index template (how we can do while creating the template). By default it is getting enable while creating the template.

Thanks,  
Debasis

[Previous page](https://discuss.elastic.co/t/automatic-index-creation/361788.md?page=3)
