# Logstash and auto-index creation with ECK

**URL:** <https://discuss.elastic.co/t/logstash-and-auto-index-creation-with-eck/360391>\
**Category:** Elastic Cloud on Kubernetes (ECK)\
**Tags:** elastic-stack-security\
**Created:** [May 28, 2024, 3:03pm UTC](https://discuss.elastic.co/t/logstash-and-auto-index-creation-with-eck/360391 "2024-05-28T15:03:02Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![durera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/durera/32/134874_2.png) [@durera](https://discuss.elastic.co/u/durera)\
**Post date:** [May 28, 2024, 3:03pm UTC](https://discuss.elastic.co/t/logstash-and-auto-index-creation-with-eck/360391/1 "2024-05-28T15:03:02Z")

</div>

I've been looking at replacing our hoomebrew stack with ECK this last week, and have the basic flow from filebeat -\> logstash -\> elasticsearch working. It's really impressive how it all just wires up together, and the quickstart docs made it a breeze to set up.

I'm now looking at more advanced (relatively) configuration such as controlling the indices used in the logstash output and I don't know whether I'm hitting a bug, trying to do something that's just not supported with ECK, or is just simple user error.

OOTB logstash was writing to an index named `logs-generic-default`, and I figured to change this all I needed to do was:

```auto
      output {
        elasticsearch {
          ...
          index => "mas-%{+YYYY.MM.dd}"
        }
      }

```

However, this throws up an error because ECK hadn't wired up things such that Logstash can manage indices.

```auto
error=>{"type"=>"security_exception", "reason"=>"action [indices:admin/auto_create] is unauthorized for user [eck-mas-eck-mas-logstash-user] with effective roles [eck_logstash_user_role] on indices [mas-2024.05.28], this action is granted by the index privileges [auto_configure,create_index,manage,all]"}}

```

I assume this is something it can be configured to do, but I have had no success working out **how**.

When I lookup the mentioned role and user in elasticseaarch (`_security/role`, `_security/user` APIs) to see what permissions they had been set up with I get back zero users and 30 roles, none of which are named `eck_logstash_user_role`. Which is confusing, because it was managing to successfully put the data into that default index.

To get around this I replaced the QS\_ES\_USER, QS\_ES\_PASSWORD env var substitutions in the output config with the main `elastic` username/password and it successfully set up the new index as expected

```auto
        elasticsearch {
          hosts => ["${MAS_ES_HOSTS}"]
          user => "elastic"
          password => "xxxxxxxxxxxx"
          ssl_certificate_authorities => "${MAS_ES_SSL_CERTIFICATE_AUTHORITY}"
          index => "mas-%{+YYYY.MM.dd}"
        }
      }

```

Obviously this is just a workaround, I'd like to understand how this is meant to work properly. I couldn't find anything searching these forums, the GitHub issues, or the docs explaining how to set things up such that Logstash would be able to create indexes on-demand, and the missing user & role just added to my confusion.

Any tips/pointers very much welcomed!

---

<div class="post-metadata">

**Author:** ![yago82](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yago82/32/97755_2.png) [@yago82](https://discuss.elastic.co/u/yago82)\
**Post date:** [May 29, 2024, 8:52am UTC](https://discuss.elastic.co/t/logstash-and-auto-index-creation-with-eck/360391/2 "2024-05-29T08:52:37Z")

</div>

Hi,

You can create a custom role in Elasticsearch that has the necessary permissions to create indices. The role should have the manage or create\_index privilege at the index level.

Create a new logstash\_index\_writer user.

Once the role is created, you can assign it to the Logstash user.

Regards

---

<div class="post-metadata">

**Author:** ![orion-ua](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/orion-ua/32/122030_2.png) [@orion-ua](https://discuss.elastic.co/u/orion-ua)\
**Post date:** [July 10, 2025, 12:35pm UTC](https://discuss.elastic.co/t/logstash-and-auto-index-creation-with-eck/360391/3 "2025-07-10T12:35:03Z")

</div>

@durera

Seems you need to extend permissions for the `eck_logstash_user_role`.  
Here is an [example](https://raw.githubusercontent.com/elastic/cloud-on-k8s/3.0/config/recipes/logstash/logstash-es-role.yaml) how to do that.

Here is my example with extended permissions:

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: roles
stringData:
  roles.yml: |-
    # Role for ECK Logstash user
    # https://www.elastic.co/docs/deploy-manage/deploy/cloud-on-k8s/configuration-logstash#k8s-logstash-pipelines-es
    eck_logstash_user_role:
      # Default privileges
      cluster: ['monitor', 'manage_ilm', 'read_ilm', 'manage_logstash_pipelines', 'manage_index_templates', 'cluster:admin/ingest/pipeline/get']
      indices:
      - names: ['logstash', 'logstash-*', 'ecs-logstash', 'ecs-logstash-*', 'logs-*', 'metrics-*', 'synthetics-*', 'traces-*']
        privileges: ['manage', 'write', 'create_index', 'read', 'view_index_metadata']
      # Additional privileges to read and write logs
      - names: ['mycompany-logs*']
        privileges: ['read', 'write', 'view_index_metadata']

---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
spec:
  version: 8.17.0
  auth:
    roles:
      - secretName: roles
  nodeSets:
    - name: default
      count: 3
      config:
        node.store.allow_mmap: false

```