Hello, let's imagine that I have a cluster with 5 nodes and each node is a client, I want these 5 to divide the data from these clients into spaces, I would also like to install packebeat and filebeat on each node and send the data of these beats to the client's index and then load that data into the client's space. I also wanted to load the alerts from the security tab and based on the information in each index I wanted there to be alerts.
I think that the question about alerts is already answered in your previous post with the same name.
This question seems to be what architecture you should use, I suggest that you edit the name of your topic to reflect this and so this question is not seen as a duplicate of your other question.
Yes that's what I want to know. What would be the best architecture.
First thing is, to have a resilient cluster you need 3 master nodes, and it is recommended to have dedicated master nodes, so with 5 nodes you would have 3 master nodes and 2 data nodes.
Second, if you want to have a data node per tenant (which you called client here), you will have a lot of work to do, you will need to heavily use the shard allocation awereness, basically you will create a custom attribute, for example tenant, and each data node will have a different value for this attribute, then you will need to configure this in every index template for every tenant.
Depending on the number of clients this can become a nightmare to manage.
Also, where the node will be located? On your infrastructure on your client (tenant) infrastructure?
To do this you would need to follow this documentation on how to create a custom name for each index.
This was already answered in your previous post.
When you say client, do you mean Elasticsearch one or a business client (ie one of your customers)?
I want all customer information to be located on the client infrastructure. Also is there any documentation about how to configure the index template with the attribute ?
This is not recommended, you would have a Elasticsearch cluster with nodes in different physical locations, an issue on a node of your customer could impact your entire cluster and all your other customers.
Also, how would you connect the nodes on different physical infrastructure? Would you create a VPN between your infrastructure and each one of your clients? Would you expose your cluster on the internet? There are so many issues with this infrastructure.
I'm yet to see these issues. My focus now is to filter the data. My focus now is to monitor different nodes and send the information to separate indexes and then analyze these indexes.
Did you check the documentation and changed your configuration?
First you will need to add a custom attribute to every data node and restart the node, as explained in the documentation.
For example, you will need to add something like this:
node.attr.customer_name: customerName
Then you will need something like this in the elasticsearch.yml of every master-eligible node:
cluster.routing.allocation.awareness.attributes: customer_name
Then you will need to create a template to add the routing settings in every index for every customer, to create a template you need to check this documentation.
A simple template would be like this:
{
"index_patterns" : [ "customername-*"],
"priority": 1,
"version": 1,
"template": {
"settings" : {
"index" : {
"routing" : {
"allocation" : {
"require" : {
"customer_name" : "customerName"
}
}
}
}
}
}
}
This will add the setting index.routing.allocation.require.customer_name with the value customerName in every indice that matches the index pattern customername-*, so you would need to configure your filebeat and packet beat to write to customername-filebeat-* and then it will be written in this data node.
With these settings I would store all beat data in that index right? How would I then receive alerts depending on the index data?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.