Connecting DMZ ENV to ELK in the LAN

[gilisade]

Hi there
we have an ELK STACK in our LAN env (internal environment)
we also have a DMZ env in the organization
we want to install beats (metric winlog file...) on every server in the DMZ env
and we want the output to get to the ELK in the LAN
i read that you can use redis for that ( and configure in the beat YAML a redis output)
but i also read that you can use reverse proxy - BTW - how can we set that up?

are there any other options?
what is the best one?

BTW - we plan on installing X-PACK license soon

[ruflin]

You probably have different options here. If you write to redis or kafka first, you can use Logstash from your LAN to access redis / kafka to fetch the data. If the reverse proxy setup with the elasticsearch output will work perhaps @steffens can give some more details?

[steffens (Steffen Siering)]

I'd prefer a queueing system like redis/kafka to have a separate single point of contact between the subnetworks. But a reverse proxy should work as well. Advantage of queuing system is, if ES is not reachable, data can still be pushed to queue.

Given you're having 2 separate networks, some consideration about access rights to queueing system or reverse proxy should taken into account.

[gilisade]

Hi
thanks @ruflin @steffens for your replies
i think the redis solution would be better for us since we can use logstash

[ruflin]

@gilisade Logstash supports Redis and Kafka. Also be aware the Logstash has since 5.1 a persistent queue itself: https://www.elastic.co/guide/en/logstash/5.1/persistent-queues.html

[gilisade]

Do you suggest we work with both redis and persistent queue ? Or push events directly to logstash with beats and use the persistent queue ?

[ruflin]

I would say in the case of Redis you can only use logstash with the persistent queue. If you need a distributed queue, I would go with Kafka and LS.

[gilisade]

What if we install logstash on the proxy server (between the dmz and the lan) beats from dmz will write to this logstash (with persistent queue) and this logstash writes directly to ES in the lan (we just open one port between logstash and ES) ?
Is that option preferable?

[steffens (Steffen Siering)]

Sounds like a valid approach.

With the proxy server being crucial for networks to operate correctly, I'd consider putting logstash on another machine within the protected network and simply forward/proxy the TCP connection.