I am trying to understand the setup of a (central) Elastic Stack cluster used by 10+ projects and 20+ applications. Our applications consist of frontend web and Java backend applications.
One Cluster
Is one central cluster (with multiple nodes) sufficient?
Managability:
not in terms of hardware capacity but;
ease of making changes to the cluster (configuration) without disturbing other users
data encapsulation (protection); some indices only visible for owners
managing pipelines
managing security
Multiple Clusters
Or should you have multiple clusters; one for each project?
If multiple clusters, how about sharing data (indices / log events) between 2 clusters?
is it possible to share indices in realtime? How does Elastic Stack support this?
I think your best option is to build multiple clusters. Data encapsulation and security is not a big thing and can be done on a single stack too (Document Level Security allows filtering data on a per-role basis, Field Level Security allows hiding sensitive fields on a per-role basis).
The main benefits for you are:
You can update the clusters independently from each other
some features(e.g. managing pipelines) require stack admin privileges. With multiple stacks the team can get those privileges on their own stack.
Data can be shared between the clusters in different ways:
Cross Cluster Search allows searching in remote clusters. As you search directly in the remote cluster the data is always up to date
Cross Cluster Replication copies the data from a single cluster to one or more other clusters. The search will only execute on your local cluster but you can only see the data that was already replicated.
There may be more options but this can give you a first overview.
This is pretty broad and depend on the changes, do you have any example? Some changes you can do without disturbing the users, other changes you may need to restart the cluster which can disturb of the user, it is pretty hard to answer without an example of what kind of changes you are referring to.
Can be done pretty easily with roles and setting a naming convention for the indices, as mentioned you also have Document and Field level security, but those are paid features, with the Basic and free license you only have index level security.
What kind of pipelines? Ingest pipelines? Pretty easy to manage as well, you have APIs or can do in Kibana UI.
Also, no issues, APIs and Kibana UI.
Would help if you provided more context what you want to achieve.
As I understand now from your and Wolfram's reply data encapsulation is not a problem. This can be arranged fairly easy with Role based security.
I am new into the Elastic Stack and just want to know what the intended way of usage is for a big company. Because in theory you can just work with one cluster if data ingestion is fairly generic for all projects/users.
At our company we make use of PaaS Cloud platform which provides all its users with a standard infrastructure, a Java Application Server and Database. Filebeat comes pre-installed on each cloud instance. Log files (infra + application logs) from all Cloud instances are sent to the same Kafka topic. Each log event gets tagged with a project specific field.
What about managing Logstash pipelines in this specific situation?
Can you have separate Logstash pipelines for each project/application/log event type?
Working with just one pipeline for 10+ users (projects) will be very difficult I guess.
What licensing level are your using? If free, you need different stacks. If you're paying for licenses, a stack per app will probably be expensive. As others have said, RBAC will provide the security you need for multi tenant stacks.
Yes, use a limited account for the logstash connection, I suggest different accounts for "writers" vs "readers" anyway. If you move to the beats/agents using ILM and streams, they need rights to define templates and indices.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.