Restrict User to a search filter

[robin.msr (Robin)]

Hi,
I have a question about Elastic best practices.

We use Elasticsearch with Kibana and Filebeat to log access logs from a loadbalancer. Behind it, we have dozens of websites. Each log contains a field which defines the backend name. Now there are external programmers, who have to get access to all logs from a specific backend.

Right now, we solved it by configuring filebeat to create a different index for each backend. So we have indices like traefik-backend1-2019.07.01, traefik-backend2-2019.07.01 and so on. With the X-Pack Security settings we now can restrict a user to a specific index pattern traefik-backend2-*. All of these indices have the same mappings, so it would be preferable to combine all backends in one index per timerange and restrict a user to a defined filter like backend_name : backend2.

As the number of backends grows, the number of open shards grows to, so with our current solution we reached the limit of 1000 shards per cluster.

What's the best way to solve this? As more backends continue to flow in, I doupt that this is the way to do. Could you mark an index as read-only to exclude it from open shards, or have it to be closed? As far as I know, you can't query over closed inices.

Thanks in advance

[Christian_Dahlqvist]

Document level security will allow you to filter documents based on a field like you describe, but this feature in not available at the basic license level. As I assume your indices may be quite small you might be able to get the current approach to go s bit further by switching to e.g. monthly indices and make sure each index has 1 primary shard.

[robin.msr (Robin)]

Thank you for your quick reply. As the indices are indeed quite small, I changed them to monthly rounded time. The downside is, that I have to set the delete phase as a multiple of months, so my previous rule "delete after 14 days" doesn't work anymore. But thats okey for the moment.

Thanks for your help.