Elasticsearch Architecture

I have set up an Elasticsearch cluster to handle logs from a Kubernetes (K8s) cluster hosting over 500 applications. Retention policies need to be configured per application, with a default retention of 20 days and extended periods for specific applications.

The Elasticsearch cluster comprises 9 nodes (16 vCPUs, 64.0 GiB per node): 5 master+hot nodes and 4 data nodes. Each application has a dedicated index, resulting in over 500 indices daily and a total log size of approximately 1.5 TB per day.

Logs are initially written to the hot nodes and, using Index Lifecycle Management (ILM), are moved to the data nodes after one day.

Currently, with this setup, I am ending up with 11K indices and over 25K shards. Additionally, I frequently encounter Elasticsearch rate limit issues. Is there a better way to architect this scenario to improve performance and scalability?

HI @rolindroy,

Welcome! Can you share your ILM policy? How long are you hanging onto the data in hot storage? Are you using cold, frozen, delete stages?

Can you also share your shard settings (number of shards from the index template)?

There is some guidance here in the docs on sizing that may help as well.

Hello @carly.richmond, thank you for responding. We have only hot and warm tiers. All the data is written to the hot tier, moved to the warm tier after a day, and deleted after 20 days. Additionally, we take a snapshot to S3 every day.

Here is the ILM policy:

  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "1d",
        "actions": {
          "set_priority": {
            "priority": 50
          },
          "allocate": {
            "number_of_replicas": 1
          }
        }
      },
      "delete": {
        "min_age": "20d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

Index Template:

{
  "index": {
    "lifecycle": {
      "name": "common-lifecycle-policy"
    },
    "routing": {
      "allocation": {
        "include": {
          "_tier_preference": "data_hot"
        }
      }
    },
    "refresh_interval": "5s",
    "number_of_shards": "2"
  }
}

HI @rolindroy Welcome to the community

^^^^ This is your issue and not really a best practice ....

Why?

So Assuming the 1.5TB is Primary

(1500GB / Day) / 500 Apps / 1 Day = 3GB indices and assuming 1 Primary Shard, 3GB Shards

This means many, many small indices shards, which cause the index and shards explosion (many) and, of course, is much less performant for searches.

We recommend shards between 30 and 50GB... 30GB is a good sweet spot...

SO what does that mean...

If you want to move to Warm each Day.... you should stop partitioning by App... Or By Each App...

For example,

  • You could put All Apps in a Single Index,
  • Figure your Set Your Shards to 30GB - 50GB
  • Set the number of Primaries to 5 so each index ends up about 150GB - 250GB
  • And you roll over about 10-5 Times a Day...
  • Put A term that represents the App, and then, if you still want simple segmentation, you can use filtered aliases.

This would significantly reduce the number of indices / shards etc... better performance etc

This is all theory until you test it :slight_smile:

1 Like