Below is the configuration I have for my 2 node elasticsearch cluster.
Pls let me know how I should configure the logstash elasticsearch output. Should the data be sent to both the below nodes?
I am fully aware that a 2 node setup is not the best way forward, but my company has resource constraints hence this is the best we can setup as of now.
Hi,
Yes I would recommend to configure the output to both nodes.
I would also recommend to have a look at ilm (index lifecycle management).
The output configuration depends for each infrastructure. But in the documentation you can find multiple examples how to configure (with ilm/ without ilm, with ssl/ without ssl, etc.)
You already have configured the 2 nodes to be both, data and Master node.
So the only thing I further could recommend is to set the replica of all indexes to 1 (Default setting).
Ok, when the data the sent to only one node in a cluster setup it gets replicated on the other node right? Why do we then need to send data to both nodes? @KoettingSimon
You do not send it to both nodes, you send it to either node, which allows you to load balance. If you had more than 2 nodes this is how you achieve high availability.
Also, regarding encrypted communications between both the elasticsearch nodes. Do i need to generate 2 separate ssl certificates and add it this way -
ssl cert of node1 in elasticsearch.yml of node2 AND
ssl cert of node2 in elasticsearch.yml of node1
OR
just a single ssl certificate which contains domain names of both node1 and node2.
I currently have kibana installed on node2 and have the domain name as node2.org, so this should point to node1 as well right?
As Christian said, you do not send the data to both nodes simultaneously. If you configure multiple hosts in the output configuration Logstash will load-balance between the host.
Sets the host(s) of the remote instance. If given an array it will load balance requests across the hosts specified in the hosts parameter.
So you dont need to configure the output to both host, but i recommend it to use the load balancing.
As i can see in the config you posted, both nodes are masters, so which node do you mean?
You need to create a certificate for each node and configure in the elasticsearch.yml the certificate of the node, so on node1 you configure the node1 certificate.
Each node need to trust the issuing CA.
To get started with Elastic-Stack security i recommend this blog article.
Also when i hit the kibana link the request would go to either node1 or node2 if i configure both in logstash output (since that would make it load balanced) ?
You need to create a certificate for each node and configure in the elasticsearch.yml the certificate of the node, so on node1 you configure the node1 certificate.
Each node need to trust the issuing CA.
Since my setup will be load balanced now, do i still have 2 get 2 separate certificates or can i include both node1 and node2 domains in the single certificate. I have raised a request for a single cert for node2, just thinking if i should include node1 in it or raise a separate cert for node1.
As i can see in the config you posted, both nodes are masters, so which node do you mean?
I meant node2 but now i will add both nodes in my logstash output
Im not an SSL-Pro but as i can read here this should be possible.
But why do you want to use the same certificate? I would recommend to generate seperate certificates.
Hi @KoettingSimon as suggested I have got separate ssl certificates for node1 and node2 and I have configured node1 ssl certificate in node1 and likewise for node2.
However, I am receiving the below error now:
[server] failed to establish trust with server at []; certificate is not trusted in this ssl context ([xpack.security.transport.ssl])
Sounds like you trying to access via the IP and SSL but in the Certificate the IP is not included as a subject alternative name.
Some infos about the subject alternative name you can find here.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.