I have a question about Elastic best practices.
We use Elasticsearch with Kibana and Filebeat to log access logs from a loadbalancer. Behind it, we have dozens of websites. Each log contains a field which defines the backend name. Now there are external programmers, who have to get access to all logs from a specific backend.
Right now, we solved it by configuring filebeat to create a different index for each backend. So we have indices like
traefik-backend2-2019.07.01 and so on. With the X-Pack Security settings we now can restrict a user to a specific index pattern
traefik-backend2-*. All of these indices have the same mappings, so it would be preferable to combine all backends in one index per timerange and restrict a user to a defined filter like
backend_name : backend2.
As the number of backends grows, the number of open shards grows to, so with our current solution we reached the limit of 1000 shards per cluster.
What's the best way to solve this? As more backends continue to flow in, I doupt that this is the way to do. Could you mark an index as read-only to exclude it from open shards, or have it to be closed? As far as I know, you can't query over closed inices.
Thanks in advance