EDIT: While writing this I found the problem: The setting unmapped_groups_as_roles: true in the elasticsearch.yml caused the problem. I don't know why this happened, but after setting it to false, everything works like expected.
This is the same issue as found by another user - I promise I searched for the issue in existing topics, but it was typical that the I found another thread 30 mins after posting my own (I even stole his edit - apologies and credit to nick.e Performance Issues and timeouts with Elasticsearch)