How to achieve high availability of ELK Stack?

Elastic Stack Logstash
1

Hi Experts,

Sorry if I am asking this question in wrong group. I have Archsight CEF data which I will be ingesting into ES using LS . To achieve HA I have total 8 machines 2-ES, 2-LS(these also have syslog-ng), 2-Kibana and 2-Nginx(for reverse proxy and load balance).

Now, In the above architecture I am not able to distribute traffic equally between two syslog-ng/LS machines in round-robin method from Nginx. So below is what I want to achieve
CEF DATA -> NGINX INSTANCE ----REDIRECT(load balance) ---> syslog-SERVER 1(LS1) AND syslog-SERVER 2 (LS2).

Can someone help me to achieve highly available ELK stack without duplicating data.I already checked https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html but no luck yet .

Regards
Vikas

2

Now, In the above architecture I am not able to distribute traffic equally between two syslog-ng/LS machines in round-robin method from Nginx.

Why not? What's the problem?

3

In order to operate properly, Elasticsearch need an elected master node in the cluster. If this is not available the cluster is not able to accept writes as this could lead to data loss. Masters are elected by a majority of master eligible nodes, which means that you will need at least 3 Elasticsearch nodes in order to achieve HA.

4

I know this is something to do with Nginx but while sending data to nginx on port 9516 I am getting below error .

<html>
<head><title>400 Bad Request</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx/1.4.6 (Ubuntu)</center>
</body>
</html>

here is my nginx.conf

upstream syslog-ng {
syslog-ng-server1:9516;
syslog-ng-server2:9516;
}

server { # simple load balancing
listen 9516;
server_name nginxIP;
location / {
proxy_pass http://syslog-ng;
}
}

Would really appreciate if I can get any help on nginx conf .

5

Thanks Christian,

I will take care of this .Actually problem is with data loadbalance . I am in good position with ES.

Regards
VG

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.