Need Help: Date-wise Index Creation with Application Using Built-in Elasticsearch and Static Index Name

Elastic Stack Elasticsearch
,
1

## Environment Details ###

  • Elasticsearch (self-managed cluster on AWS)
  • 4 Data Nodes (1 TB each)
  • 2 Master Nodes
  • Daily data ingestion around 300 GB (without replica) and approximately 600 GB with 1 replica
  • Data source is an application that has built-in Elasticsearch integration

Current Architecture

Application (with built-in Elasticsearch integration)
↓
Elasticsearch Cluster
↓
Index: greenzone-aws_transactionalevents

  • The application pushes data directly into Elasticsearch
  • Elasticsearch connection is configured from the application UI
  • Index name is defined in the application configuration
  • The index name remains static

Problem Statement

The issue I am facing is as follows:

  1. The application allows configuring only a static index name such as:
    greenzone-aws_transactionalevents
  2. It does not support dynamic index naming like date-based patterns
  3. It does not automatically create or rotate indices based on date

Impact of Current Setup

Due to this limitation, the following challenges are observed:

  1. Continuous index growth
  • Around 300 GB data is ingested daily
  • The same index keeps growing without any rotation
  1. No date-wise data separation
  • All data is stored in a single index
  • It is difficult to isolate data for a specific day
  1. Snapshot limitations
  • Snapshots work at index level
  • It is not possible to take snapshots for specific date ranges within the same index
  1. Restore limitations
  • It is not possible to restore data for a specific date
  • Full index restore is required even for partial data
  1. Operational challenges
  • Storage management becomes difficult
  • Large index size can impact performance over time

Requirement

I want to achieve the following:

  1. Date-wise index creation, for example:
    greenzone-aws_transactionalevents-2026.04.22
    greenzone-aws_transactionalevents-2026.04.23
  2. Better data organization
  • Logical separation of data per day
  1. Granular backup and restore
  • Ability to take snapshots and restore specific date-based data

Questions

  1. Does an application with built-in Elasticsearch integration typically support:
  • Dynamic index naming
  • Date-based index creation or rotation
  1. Is there any configuration in Elasticsearch that can:
  • Automatically split or manage data by date when a static index name is used
  1. Can index templates, ingest pipelines, or any Elasticsearch feature:
  • Override or influence index naming when data is pushed from an external application
  1. What are the best practices for handling:
  • High-volume ingestion (around 300 GB per day)
  • When the source system enforces a static index name

Summary

  • The application pushes data to Elasticsearch using a static index name
  • Dynamic or date-based index naming is not supported at the application level
  • Need guidance on achieving day-wise index structure for better data management, backup, and restoration

Looking for suggestions and best practices to handle this scenario effectively.

2

The index naming is up to the application, it needs to be implemented in the applicaiton code, the roration should be controlled by an Index Lifecycle Policy or if the application changes the name each day.

I would also say that rotation based on data is not a good approach, you should aim to rotate based on size of the index.

With an index lifecycle policy you can configure it to rotate based on size and age, but not on exact dates.

An Index lifecycle policy can do that as it will have a write alias, which is static, and it uses backing index to write the data, but as mentioned, those indices will rotate based on size or age.

Yes, you can configure an index.final_pipeline in your index template so it would always call an ingest pipeline in elasticsearch before indexing, in this indexing pipeline you could use some processors to change the index name, but this would depend on your data.

The common approach is to use a data stream and rotate by size, not by day, it is easier to manage.

But you need to provide more context about this application, does it only write to elasticsearch or does it read from elasticsearch as well? Does it update the data once it is written?

3

What version Elastic are you on?

4

All components on 9.1.3

Also please give documentation if any for such cases.

5

If you have a static index name in the application

I would create a data stream template that matches that that index name

Create a custom ILM policy applied to the data stream template

Then your application will be writing to a data stream and then all the naming will be taken care of for you

And it will use the ilm policy that applied.

All the actual backing indices will be managed by the data stream.