Seeks advice on designing an efficient and reliable Elastic cluster setup, specifically regarding the placement of Logstash and Kibana nodes within a firewall-based architecture

[Weeltin]

Hi everyone,

I'm in the process of setting up an Elastic cluster, and I'm looking for some advice on the best way to proceed to ensure I end up with an efficient and reliable solution.

Here's my current setup:

My idea is to have two Log stash nodes, one on each side of the firewall. The node on the right would collect all logs from servers, clients, and network equipment. It would then forward the logs to the node on the left side, which would process the data and add it to Elasticsearch.

Does this sound like a good approach? Are there any potential pitfalls I should be aware of?

Additionally, I'm unsure about the optimal placement for 1 or 2 Kibana nodes and a fleet server. Where would be the best location to deploy them for maximum performance and accessibility, but still have it relative secure?

Any tips, feedback, or suggestions would be greatly appreciated!

Thanks in advance for your help!

[leandrojmp (Leandro Pereira)]

Hello and welcome,

@Weeltin

Does this sound like a good approach? Are there any potential pitfalls I should be aware of?

If you have only one logstash in each side, you have a single point of failure, you may need to have more logstash nodes depend on the amount of data, which can lead to the need of having a load balance.

@Weeltin

Additionally, I'm unsure about the optimal placement for 1 or 2 Kibana nodes and a fleet server. Where would be the best location to deploy them for maximum performance and accessibility, but still have it relative secure?

It is not clear where this firewall is located, if it is outside of your private network or if it inside your private network and you are using it to control the access to your cluster.

Where Kibana and Fleet will be located depends on what your clients will be, but you may put everything on the Elasticsearch side and configure the necessary firewall rules.

Also, if you install an agent on a server, this agent needs to connect to the Fleet Server, and the Fleet Server also needs to connect to Elasticsearch.

[Weeltin]

Thanks, and sorry for the late reply. had a few days off.

@leandrojmp

If you have only one logstash in each side, you have a single point of failure, you may need to have more logstash nodes depend on the amount of data, which can lead to the need of having a load balance.

I’m aware of this, and it’s something I’ll be working on addressing. However, management is currently reluctant to allocate the resources needed for additional nodes, and they’re even less inclined to approve a load balancer.

@leandrojmp

It is not clear where this firewall is located, if it is outside of your private network or if it inside your private network and you are using it to control the access to your cluster.

The firewall is located within our private network and serves to protect the Elasticsearch cluster. I have full control over this firewall since I am the one configuring it.

@leandrojmp

Where Kibana and Fleet will be located depends on what your clients will be, but you may put everything on the Elasticsearch side and configure the necessary firewall rules.

Also, if you install an agent on a server, this agent needs to connect to the Fleet Server, and the Fleet Server also needs to connect to Elasticsearch.

The plan is to install a Fleet server and agents, at least on our server.
I just need to do some additional research on that part.
For example, I need to know:

Which ports it uses?
The location where the agent is installed?
Is traffic is encrypted?
and so on..