Hi,
I plan to deploy Elasticsearch for a production environment following this architecture :
node.roles: [ master ])node.roles: [ data_hot ])node.roles: [ data_warm ])node.roles: [ data_cold])My question is : do I also need "data", "data_content", "ingest" and "transform" node roles and on which instance do I need to set these?
Thanks a lot for your help!
Hi,
I encountered a problem straight from the beginning.
After the installation of my first dedicated Master node (node.roles: [ master]) my cluster health seems red as I'm unable to reset password or generate token for adding new nodes.
So for the second node, I've setup the Hot Data node (cluster.name: bla, node.roles: [ data_hot, ingest, transform ], cluster.initial_master_nodes: ["master-1"]) without a token but this doesn't solve the problem...
So do I need to setup as the first node a classic elasticsearch without defining node.roles then when the whole architecture is deployed reconfigure it to a master only? But I will loose shard no?
Thanks for your help.
Regards.
The data role is a generic role the can act as all the other data roles, since you are planning to have tiered data nodes, you do not need the generic one, but you need a data_content node, data_hot and data_content are required.
You may add the data_content role to the data_hot node.
Also, with just one node for each data tier you won't be able to have replicas.
Since you are planning to use fleet you also need to have an ingest node and a transform node.
You will probably need to change the roles of your data_hot node to this:
node.roles: [data_hot, data_content, ingest, transform ]
You need the data_content role, as explained it is required, the system indices uses this role, this is also an important note in the documentation.
Hi @leandrojmp,
Many, many thanks for your advices and this explaination, that makes perfect sense!
I have one more question please :
data_content and data_hot roles), how to enroll in my cluster the Data_Hot node without the ability to generate an enrollment token ?Have a great day!
Regards.
Did you configure all nodes and start them or you just configured one node and started it? You need to configure your 3 masters + your data hot node and start them.
Hi @leandrojmp,
Once again many thanks for the advice!
What I was doing before :
What I am actually doing now :
cluster.name: MyClusterProd
node.name: master-1 # Change for each master node
node.roles: [ master ]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.74.30 # Change for each master node
http.port: 9200
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["master-1", "master-2", "master-3"]
http.host: 0.0.0.0
transport.host: 0.0.0.0
cluster.name: MyClusterProd
node.name: datah-1
node.roles: [ data_hot, data_content, ingest, transform ]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.74.42
http.port: 9200
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["master-1", "master-2", "master-3"]
http.host: 0.0.0.0
Despite following this documentation Bootstrapping a cluster, I don't figure out how and in which order to start each node of the cluster.
Just remember that right after the startup of master-1, the cluster health is red and I'm unable to generate an enrollment token.
Have a great day.
Regards.
I never used this enrollment token approach, but if I'm not wrong it works only on specfic scenarios, like your nodes will have all the roles and the certificate will be auto-generated.
The documentation for the bootstrapping a cluster assumes that your nodes will have all roles, so you will already have a bootstrapped cluster and could use the enrollment token, this is not your case since you want to have master dedicated and data dedicated nodes and you need nodes with master, data_content and data_hot roles.
My suggestion would be to configure your data_hot/data_content nodes to also be master nodes and start all the 5 nodes at the same time and wait for the cluster formation.
After that you could remove the master role for the data nodes and restart them.
Thanks a lot for the suggestion about giving to Data Hot node the master role for the initial cluster setup then remove it.
Could you please provide me what enrollment approach do you usually use?
Thanks!
Basically you just need to create the certificates for all nodes and configure the elasticsearch.yml files, from what I understood the enrollment token only serves to simplify the certification generation, that's why it rely on you using the auto-generate certificates.
If you manually create certificates for your nodes, you won't need to use the enrollment token command, you just create the certificates, copy it to the nodes, configure the elasticsearch.yml and start the nodes.
Hi @leandrojmp,
Thanks a lot for this advice, it easily solved my problem!
Now my cluster is up and running with 3 dedicated master nodes.
Have a great day.
Regards.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.