# Automatic Index Creation

**URL:** <https://discuss.elastic.co/t/automatic-index-creation/361788>\
**Category:** Elasticsearch\
**Created:** [June 20, 2024, 11:06am UTC](https://discuss.elastic.co/t/automatic-index-creation/361788 "2024-06-20T11:06:23Z")\
**Posts on this page:** 20\
**Page:** 3

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [July 5, 2024, 6:11am UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/41 "2024-07-05T06:11:43Z")

</div>

Thanks @leandrojmp for your help. I am now able to create Index accordingly by reading the timestamp value. Could you please help me to understand `date_rounding` parameter in `date_index_name` processor.

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\
**Post date:** [July 5, 2024, 10:27am UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/42 "2024-07-05T10:27:06Z")

</div>

Hi @Debasis_Mallick,

Do you have a particular question on concern on the `date_rounding` parameter? The possible values and a description of the options you can round the date in the index name with [is present in the documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/date-index-name-processor.html).

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [July 10, 2024, 10:49am UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/43 "2024-07-10T10:49:24Z")

</div>

@leandrojmp Now we are able to create automatic Index based the timestamp value mentioned in the record. But I want to incorporate certain additional settings for these Index (example: If want to change datatype of fields, replicas etc ..) . How we achieve the same with this dynamic Index.

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [July 12, 2024, 7:42am UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/44 "2024-07-12T07:42:35Z")

</div>

Hi @leandrojmp Any help on the above request.

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [July 12, 2024, 8:25am UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/45 "2024-07-12T08:25:11Z")

</div>

If you want to control index mappings you need an index template. If you need to change mappings you will need to reindex into a new index with the desired mappings.

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [July 15, 2024, 6:00am UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/46 "2024-07-15T06:00:27Z")

</div>

But in below pipeline the index is getting created dynamically so where to mention the index template so that it create with proper settings.

```auto
PUT _ingest/pipeline/parse_elastic_data_v2
{
  "processors": [
    {
      "csv": {
        "description": "Parse elastic Data From CSV Files",
        "field": "message",
        "target_fields": ["sequence",
        "component","tenant",
        "service_id","session_id",
        "timestamp","edr_version"],
        "separator": ",",
        "ignore_missing":true,
        "trim":true
      }
      },
	  { "date": {
          "field": "timestamp",
          "target_field": "timestamp_em",
          "formats": ["UNIX_MS"]
        }
	     },
		 
      "date_index_name": {
        "field": "timestamp",
        "index_name_prefix": "sfw-",
        "date_rounding": "d",
        "date_formats": ["UNIX_MS"]
      }
    }
 ]
}

```

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\
**Post date:** [July 15, 2024, 7:10am UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/47 "2024-07-15T07:10:56Z")

</div>

There is the `index_patterns` option in the index template that accepts wildcards so you can specify expressions that match your template:

> **[Create or update index template API | Elasticsearch Guide \[8.14\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-template.html)**

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [July 15, 2024, 10:58am UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/48 "2024-07-15T10:58:02Z")

</div>

@carly.richmond As I mentioned above , the pipeline is creating a dynamic Index starting with `sfw-*` as per pipeline `index_name_prefix`. How we can add Index template in the pipeline. Please advise .

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\
**Post date:** [July 15, 2024, 11:36am UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/49 "2024-07-15T11:36:20Z")

</div>

Perhaps I'm overly-simplifying things, but would you not specify prefix `sfw-*` as your index pattern in your index template? If not can you explain why you can't do that?

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [July 15, 2024, 3:01pm UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/50 "2024-07-15T15:01:12Z")

</div>

@carly.richmond Below is my index template and pipeline already mentioned above. The template name I used in filebeat.yml files but still Index are not getting created with the settings what I provided. Could you please help if I am missing anything.

```auto
setup.template.name: "sfwidx_template"
setup.template.enabled: true

```

```auto
{
  "template": {
    "settings": {
      "index": {
        "number_of_replicas": "1",
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_content"
            }
          }
        }
      }
    },
    "mappings": {
      "properties": {
        "action_id": {
          "type": "integer"
        },
        "component": {
          "type": "integer",
          "ignore_malformed": false,
          "coerce": true
        },
        "edr_version": {
          "type": "keyword"
        },
        "sequence": {
          "type": "keyword"
        },
        "service_id": {
          "type": "integer"
        },
        "session_id": {
          "type": "keyword"
        },
        "tenant": {
          "type": "keyword"
        },
        "timestamp": {
          "type": "date"
        }
      }
    },
    "aliases": {
      "sfwindex": {}
    }
  }
}

```

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [July 16, 2024, 6:59am UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/51 "2024-07-16T06:59:45Z")

</div>

@carly.richmond Any help on this.

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\
**Post date:** [July 16, 2024, 9:49am UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/52 "2024-07-16T09:49:13Z")

</div>

Hi @Debasis_Mallick,

I'm not sure I follow what you mean. Index templates tie to the index pattern (so a pattern matching the index name) rather than a pipeline. I believe you said you were new to Elastic so I would recommend having a look at the [documentation on index templates](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html) if you haven't already.

Can you confirm that you've specified the ingest pipeline in your filebeat configuration [as covered here](https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ingest-node.html)? If so, once the index template has also been created you should be reindexing/ reingesting the data using the pipeline to make sure the index template is being applied.

If you think the index is not being applied but it should, you could see if the [`simulate_index` API](https://www.elastic.co/search-labs/blog/elasticsearch-index-template#determining-which-template-an-index-will-use-when-it-is-created) can help you figure out if the template is being applied or not.

Let us know if that helps!

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [July 16, 2024, 12:08pm UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/53 "2024-07-16T12:08:54Z")

</div>

Hi @carly.richmond  
Below two are my requirement.

1. We had implement Filebeat to read the CSV file and generate an index based on the timestamp field within each CSV record dynamically. For instance, if the timestamp (in epoch format) is **1688927400000** , the index created will be named `sfw-09-07-2023`. This functionality is successfully implemented using the pipeline `date_index_name`.

2. As the index is generated dynamically based on the timestamp column value, we encounter a limitation in customizing the data type for fields within the index `sfw-09-07-2023`. Currently, all fields default to the keyword data type. We are seeking guidance on how to address this issue.

As per your suggestion to use Index templates, we have not been able to resolve this matter.

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 16, 2024, 12:29pm UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/54 "2024-07-16T12:29:54Z")

</div>

I think that you need to load the template manually, not using the `setup.template.*` options from filebeat.

Those options expect your index name to follow a pattern like this:

```auto
filebeat-FILEBEAT_VERSION

```

You can have a custom index name, but it would need to follow the same pattern, something like `custom_name-8.14.3` for example.

So in your case you will need to create your template and load it manually as mentioned in the documentation shared.

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [July 16, 2024, 2:57pm UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/55 "2024-07-16T14:57:46Z")

</div>

@leandrojmp When I query template , we got the details. Can you help me with the command , how to load the template manually. Sorry to bother you with a simple question, as I am new to Elasticsearch.

`GET _index_template/sfwidx_template`

```auto
{
  "index_templates": [
    {
      "name": "sfwidx_template",
      "index_template": {
        "index_patterns": [
          "sfw-"
        ],
        "template": {
          "settings": {
            "index": {
              "number_of_replicas": "1"
            }
          },
          "mappings": {
            "properties": {
              "sequence": {
                "type": "keyword"
              },
              "component": {
                "coerce": true,
                "index": true,
                "ignore_malformed": false,
                "store": false,
                "type": "integer",
                "doc_values": true
              },
              "edr_version": {
                "eager_global_ordinals": false,
                "norms": false,
                "index": true,
                "store": false,
                "type": "keyword",
                "index_options": "docs",
                "split_queries_on_whitespace": false,
                "doc_values": true
              },
              "action_id": {
                "type": "integer"
              },
              "service_id": {
                "type": "integer"
              },
              "session_id": {
                "type": "keyword"
              },
              "tenant": {
                "type": "keyword"
              },
              "timestamp": {
                "type": "date"
              }
            }
          },
          "aliases": {
            "sfwindex": {}
          }
        },
        "composed_of": [],
        "_meta": {
          "description": "sfw Index template"
        }
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [July 22, 2024, 10:59am UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/56 "2024-07-22T10:59:58Z")

</div>

@leandrojmp I had tried below command to load Index template manually but getting error. Could you please just verify if the syntax is correct or not.

```auto
[root@cb-4 ~]# filebeat setup --index-management -E setup.template.name="sfwidx_template" -E setup.template.pattern="sfw-*" -E 'output.elasticsearch.hosts=["10.10.18.59:9200"]' -E setup.ilm.overwrite=true
Exiting: error loading template: failed to put data stream: could not put data stream: 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"no matching index template found for data stream [sfwidx_template]"}],"type":"illegal_argument_exception","reason":"no matching index template found for data stream [sfwidx_template]"},"status":400}. Response body: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"no matching index template found for data stream [sfwidx_template]"}],"type":"illegal_argument_exception","reason":"no matching index template found for data stream [sfwidx_template]"},"status":400}

```

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 22, 2024, 12:43pm UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/57 "2024-07-22T12:43:24Z")

</div>

Hello,

As mentioned you need to do this **manually** , not using filebeat.

You need to create and load your template as explained in this [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html).

And also disable template loading in filebeat using:

```auto
setup.template.enabled: false

```

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [July 22, 2024, 3:09pm UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/58 "2024-07-22T15:09:40Z")

</div>

> [@leandrojmp](#):
>
> As mentioned you need to do this **manually** , not using filebeat.

Could you please help me how we can load template manually. I had created the above template using kibana.

Thanks,  
Debasis

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 22, 2024, 3:15pm UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/59 "2024-07-22T15:15:46Z")

</div>

It is explained in the previous linked documentation.

What does the template you created looks like? Did you use the API through dev tools or using the Kibana UI interface?

To load it manually you juste use:

```auto
PUT _index_template/template_name
{
    "payload of your template"
}

```

After that the template is already available and will match your indices?

---

<div class="post-metadata">

**Author:** ![Debasis\_Mallick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/debasis_mallick/32/123723_2.png) [@Debasis\_Mallick](https://discuss.elastic.co/u/Debasis_Mallick)\
**Post date:** [July 23, 2024, 6:49am UTC](https://discuss.elastic.co/t/automatic-index-creation/361788/60 "2024-07-23T06:49:21Z")

</div>

> [@leandrojmp](#):
>
> What does the template you created looks like? Did you use the API through dev tools or using the Kibana UI interface

I had created using Kibana UI interface.

Thanks,  
Debasis

[Previous page](https://discuss.elastic.co/t/automatic-index-creation/361788.md?page=2)

[Next page](https://discuss.elastic.co/t/automatic-index-creation/361788.md?page=4)