# Optimizing index template for logs from Kubernetes workloads

**URL:** <https://discuss.elastic.co/t/optimizing-index-template-for-logs-from-kubernetes-workloads/386421>\
**Category:** Elastic Search\
**Created:** [May 20, 2026, 5:41am UTC](https://discuss.elastic.co/t/optimizing-index-template-for-logs-from-kubernetes-workloads/386421 "2026-05-20T05:41:46Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shammi\_ST](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shammi_st/32/143619_2.png) [@Shammi\_ST](https://discuss.elastic.co/u/Shammi_ST)\
**Post date:** [May 20, 2026, 5:41am UTC](https://discuss.elastic.co/t/optimizing-index-template-for-logs-from-kubernetes-workloads/386421/1 "2026-05-20T05:41:46Z")

</div>

Hi all, Im having the following setup for receiving logs to my Kibana from GKE hosted workloads:

Fluentbit \> Elasticsearch \> Kibana

My Elasticsearch and Kibana are running on one single server so I have a single node setup. I have a requirement where I want to look into possible methods of reducing storage consumption by the indices as my daily indices have started to increase to around 5GB in size. I already have a life cycle policy to retain logs only for the necessary time period but even with this in place it still holds a considerable amount.

I have read online that if I'm running Elasticsearch as single node it is recommended to set number\_of\_replicas to 0 instead of 1 to prevent duplication. Mine was set to the default 1 and I have configured it to 0. I was researching what else could I do to the index to optimize logging for microservices to reduce any unnecessary storage consumption while also maintaining a decent query speed. I haven't had any luck figuring things out based on already available resources. I would appreciate any support given to optimize my index.

My current index is as follows:

```auto
{
  "order": 0,
  "version": 60001,
  "index_patterns": [
    "logstash-*"
  ],
  "settings": {
    "index": {
      "number_of_shards": "1",
      "refresh_interval": "5s",
      "number_of_replicas": "0"
    }
  },
  "mappings": {
    "dynamic_templates": [
      {
        "message_field": {
          "path_match": "message",
          "mapping": {
            "norms": false,
            "type": "text"
          },
          "match_mapping_type": "string"
        }
      },
      {
        "string_fields": {
          "mapping": {
            "norms": false,
            "type": "text",
            "fields": {
              "keyword": {
                "ignore_above": 256,
                "type": "keyword"
              }
            }
          },
          "match_mapping_type": "string",
          "match": "*"
        }
      }
    ],
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "geoip": {
        "dynamic": true,
        "type": "object",
        "properties": {
          "ip": {
            "type": "ip"
          },
          "latitude": {
            "type": "half_float"
          },
          "location": {
            "type": "geo_point"
          },
          "longitude": {
            "type": "half_float"
          }
        }
      },
      "@version": {
        "type": "keyword"
      }
    }
  },
  "aliases": {}
}

```

ChatGPT provided me an optimized index as follows, is it a good replacement for my existing one:

```auto
{
  "index_patterns": [
    "logstash-*"
  ],
  "template": {
    "settings": {
      "index": {
        "number_of_replicas": 0,
        "lifecycle": {
          "name": "metrics"
        },
        "codec": "best_compression",
        "query": {
          "default_field": [
            "message"
          ]
        }
      }
    },
    "mappings": {
      "dynamic_templates": [
        {
          "match_ip": {
            "match": "ip",
            "match_mapping_type": "string",
            "mapping": {
              "type": "ip"
            }
          }
        },
        {
          "match_message": {
            "match": "message",
            "match_mapping_type": "string",
            "mapping": {
              "type": "match_only_text"
            }
          }
        },
        {
          "strings_as_keyword": {
            "match_mapping_type": "string",
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        }
      ],
      "date_detection": false,
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "data_stream": {
          "properties": {
            "dataset": {
              "type": "constant_keyword"
            },
            "namespace": {
              "type": "constant_keyword"
            },
            "type": {
              "type": "constant_keyword",
              "value": "metrics"
            }
          }
        },
        "ecs": {
          "properties": {
            "version": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        },
        "host": {
          "type": "object"
        }
      }
    },
    "aliases": {}
  }
}

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 20, 2026, 6:01am UTC](https://discuss.elastic.co/t/optimizing-index-template-for-logs-from-kubernetes-workloads/386421/2 "2026-05-20T06:01:37Z")

</div>

> [@Shammi\_ST](#):
>
> `"codec": "best_compression",`

Adding best compression is a good way to reduce storage although it tends to add some load at indexing time. This is what I would have initially suggested, as it carries little risk.

The other changes to field mappings may reduce storage but could also affect dashboards and how you query the data, so is riskier. This is something I would initially avoid and only do after thorough testing in a separate environment.

> [@Shammi\_ST](#):
>
> I have read online that if I'm running Elasticsearch as single node it is recommended to set number\_of\_replicas to 0 instead of 1 to prevent duplication. Mine was set to the default 1 and I have configured it to 0.

Elasticsearch will never allocate a replica to the same node that holds the primary so changing this will not reduce storage given that you have a single node. It will however change your indices to green status as the unallocated replicas will make then yellow. It is still a change worth making.

---

<div class="post-metadata">

**Author:** ![Asmaa\_Oufkir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asmaa_oufkir/32/31746_2.png) [@Asmaa\_Oufkir](https://discuss.elastic.co/u/Asmaa_Oufkir)\
**Post date:** [May 25, 2026, 2:13pm UTC](https://discuss.elastic.co/t/optimizing-index-template-for-logs-from-kubernetes-workloads/386421/3 "2026-05-25T14:13:29Z")

</div>

## **1. What you already understood correctly**

✅ `number_of_replicas: 0` – on a single node, replicas are indeed useless.

✅ `number_of_shards: 1` – excellent for your use case (5 GB/day). Avoids unnecessary fragmentation.

✅ `refresh_interval: 5s` – reasonable for near-real-time usage.

⚠ However, **the ChatGPT mapping is dangerous** for your use case.  
It converts all string fields to `keyword` (no full-text search) and adds `match_only_text` on `message`, which will break many Kibana visualizations.  
Don't use it as-is without thorough testing.

* * *

## **2. 4 safe and immediate optimizations**

### **✅ A. Enable `best_compression` (low risk, high gain)**

Christian Dahlqvist gave you excellent advice.  
Add this to your template for new indexes:

json

```auto
"settings": {
  "index.codec": "best_compression"
}

```

> Typical gain: **20–30% less storage** on text-heavy logs.

### **✅ B. Reduce `_source` (if you don't need reindexing or scripts)**

This is **the most underestimated optimization** on a single node.

By default, Elasticsearch stores the original JSON in `_source`.  
For large logs, you can disable compression or store only specific fields.

**Example (template):**

json

```auto
"mappings": {
  "_source": {
    "includes": ["@timestamp", "message", "host", "kubernetes.*"]
  }
}

```

Or more aggressive (if you don't use scripts or reindexing):

json

```auto
"_source": { "enabled": false }

```

⚠ Warning: no `_source` means no update/reindex, but reading still works via doc\_values.

> Gain: **30–50% space** (huge for GKE logs).

### **✅ C. Use `synthetic _source` (ES 8.4+)**

If you're on a recent Elasticsearch version:

json

```auto
"mappings": {
  "_source": { "mode": "synthetic" }
}

```

This reconstructs `_source` on the fly from doc\_values/stored fields.  
Less storage, still compatible with Kibana.

### **✅ D. Remove `norms` and `doc_values` on unused fields**

Your current template already has `"norms": false` – very good.  
Also add `"doc_values": false` for fields you don't aggregate on.

Example for a `kubernetes.labels` field not used in term aggregations:

json

```auto
"kubernetes.labels": {
  "type": "text",
  "doc_values": false,
  "norms": false
}

```

> Gain: modest (5–10%) with zero functional impact if you don't aggregate on that field.

* * *

## **3. Optimizations at the FluentBit level (before Elasticsearch)**

These are **often ignored** in forums.

### **✅ E. Filter unnecessary fields upstream**

In your `fluent-bit.conf`:

ini

```auto
[FILTER]
    Name modify
    Match *
    Remove kubernetes.annotations.*
    Remove kubernetes.labels.*
    Remove log.offset
    Remove stream

```

Typically, Kubernetes annotations double log size for no reason.

> Gain: **15–40%** depending on your workload.

### **✅ F. Enable HTTP compression in FluentBit**

Between FluentBit and Elasticsearch:

ini

```auto
[OUTPUT]
    Name es
    Match *
    Host your-es
    Port 9200
    Compress gzip

```

This also saves network bandwidth, but more importantly reduces CPU load on ES.