I've been looking at replacing our hoomebrew stack with ECK this last week, and have the basic flow from filebeat -> logstash -> elasticsearch working. It's really impressive how it all just wires up together, and the quickstart docs made it a breeze to set up.
I'm now looking at more advanced (relatively) configuration such as controlling the indices used in the logstash output and I don't know whether I'm hitting a bug, trying to do something that's just not supported with ECK, or is just simple user error.
OOTB logstash was writing to an index named logs-generic-default, and I figured to change this all I needed to do was:
output {
elasticsearch {
...
index => "mas-%{+YYYY.MM.dd}"
}
}
However, this throws up an error because ECK hadn't wired up things such that Logstash can manage indices.
error=>{"type"=>"security_exception", "reason"=>"action [indices:admin/auto_create] is unauthorized for user [eck-mas-eck-mas-logstash-user] with effective roles [eck_logstash_user_role] on indices [mas-2024.05.28], this action is granted by the index privileges [auto_configure,create_index,manage,all]"}}
I assume this is something it can be configured to do, but I have had no success working out how.
When I lookup the mentioned role and user in elasticseaarch (_security/role, _security/user APIs) to see what permissions they had been set up with I get back zero users and 30 roles, none of which are named eck_logstash_user_role. Which is confusing, because it was managing to successfully put the data into that default index.
To get around this I replaced the QS_ES_USER, QS_ES_PASSWORD env var substitutions in the output config with the main elastic username/password and it successfully set up the new index as expected
elasticsearch {
hosts => [ "${MAS_ES_HOSTS}" ]
user => "elastic"
password => "xxxxxxxxxxxx"
ssl_certificate_authorities => "${MAS_ES_SSL_CERTIFICATE_AUTHORITY}"
index => "mas-%{+YYYY.MM.dd}"
}
}
Obviously this is just a workaround, I'd like to understand how this is meant to work properly. I couldn't find anything searching these forums, the GitHub issues, or the docs explaining how to set things up such that Logstash would be able to create indexes on-demand, and the missing user & role just added to my confusion.
Any tips/pointers very much welcomed!