# ELK Satelit and Master Installation
**URL:** https://discuss.elastic.co/t/elk-satelit-and-master-installation/41312
**Category:** Logstash
**Created:** [February 9, 2016, 7:12pm UTC](https://discuss.elastic.co/t/elk-satelit-and-master-installation/41312 "2016-02-09T19:12:53Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@DanMa](https://discuss.elastic.co/u/DanMa)
#### Post date: [February 9, 2016, 7:12pm UTC](https://discuss.elastic.co/t/elk-satelit-and-master-installation/41312/1 "2016-02-09T19:12:53Z")
Hello together,
i want to make a big ELK Installation.
At the one site we want have logstash Redis and Logstash Forwarder (Satelit)
at the other site we want have Logstash Redis Logstash Elasticsearch Kibana (Master).
But what in now really know at the Satelit Site. How can i get the data from redis to Logstash Forwarder?
Greetings
Daniel
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [February 9, 2016, 7:17pm UTC](https://discuss.elastic.co/t/elk-satelit-and-master-installation/41312/2 "2016-02-09T19:17:46Z")
Are you actually talking about logstash-forwarder, a separate (deprecated) product found at [https://github.com/elastic/logstash-forwarder](https://github.com/elastic/logstash-forwarder), or Logstash _acting as_ a forwarder?
And why would you want to get data _from_ Redis _to_ Logstash/logstash-forwarder? Redis would typically act as a buffer from which Logstash in the master server would pull events.
---
### Author:  [@DanMa](https://discuss.elastic.co/u/DanMa)
#### Post date: [February 9, 2016, 7:36pm UTC](https://discuss.elastic.co/t/elk-satelit-and-master-installation/41312/3 "2016-02-09T19:36:21Z")
Between the Satelit Installation and the Master installation is a VPN Tunnel.
So if the vpn tunnel is not available at the moment then we shouldn't lose the collected logs.
In that case we need a buffer like redis between logstash to get the logs from some devices and this logs should go to redis and to logstash forwarder. If the VPN Tunnel is online the logstash forwarder should ship the data to the first logstash master installation.
greetings
Daniel
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [February 9, 2016, 7:52pm UTC](https://discuss.elastic.co/t/elk-satelit-and-master-installation/41312/4 "2016-02-09T19:52:11Z")
Ship directly to Logstash if the VPN tunnel is up and otherwise buffer in Redis? No, that doesn't make sense. Just buffer the events in Redis all the time.
---
### Author:  [@DanMa](https://discuss.elastic.co/u/DanMa)
#### Post date: [February 9, 2016, 8:17pm UTC](https://discuss.elastic.co/t/elk-satelit-and-master-installation/41312/5 "2016-02-09T20:17:05Z")
Sorry, for the misunderstanding.
Sure should redis buffer the events all the time. What i meant is that Redis is then really important when the VPN tunnel is broken. What i only should know how can i get the redis data into the logstash-forwarder?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [February 9, 2016, 8:45pm UTC](https://discuss.elastic.co/t/elk-satelit-and-master-installation/41312/6 "2016-02-09T20:45:37Z")
Logstash-Forwarder has been replaced by [Filebeat](https://www.elastic.co/guide/en/beats/filebeat/current/index.html). The normal flow in this scenario is something like this:
Filebeat ---\> Logstash ---\>Redis ----(DC Boundary)---\> Logstash ---\> Elasticsearch
If you only have file based inputs, these generally handle stoppages in the pipeline quite well as they can just stop reading until the rest of the processing pipeline clears and then continue. If you only have file inputs and will not risk losing data due to aggressive log rotation, you may be able to do without using Redis and send data directly from Filebeat to the Logstash instance in the remote DC.
Other types of inputs, e.g. inputs based on TCP and/or UDP, are often not able to stop processing without causing problems upstream or losing data. If you have these types of inputs, using a message queue like e.g. Redis for buffering is often recommended.
---
### Author:  [@DanMa](https://discuss.elastic.co/u/DanMa)
#### Post date: [February 10, 2016, 7:39am UTC](https://discuss.elastic.co/t/elk-satelit-and-master-installation/41312/7 "2016-02-10T07:39:29Z")
Ok.
But in our case we need a buffer on the satellite and the master site.
I think that is a possible Installation Setup that we have in mind:
Logstash--\> Redis --\> Filebeat/Logstash-Forwarder
---(VPN-Tunnel)---
Logstash--\>Redis--\>Logstash--\>Elasticsearch--\>Kibana
Our Problem is now, how can we get some informations from redis to Filebeat/Logstash-Forwarder.
We need also a secure communicaiton between the Filebeat/Logstash-Forwarder(Satellite) and the first Logstash (Master) with for example lumberjack.
Could you help us to get this informations or where we can find them?
I hope that describes our situation.
Greetings
Daniel
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [February 10, 2016, 8:25am UTC](https://discuss.elastic.co/t/elk-satelit-and-master-installation/41312/8 "2016-02-10T08:25:54Z")
Filebeat, and Logstash-Forwarder before it, is a data collection agent, and as such almost always places at the start of the kindest pipeline, not in the middle. Logstasg is able to read and write to/from Redis and can also connect to other Logstash instances via the lumberjack protocol, which like the beats protocol offers encryption and compression.
The architecture you describe sounds complex, and I would like to understand whether it could be simplified. Do you have inputs that are not file based? Why do you need Redis in both locations? Is it because you want t be able to buffer locally collected logs?
---
### Author:  [@DanMa](https://discuss.elastic.co/u/DanMa)
#### Post date: [July 5, 2016, 9:01am UTC](https://discuss.elastic.co/t/elk-satelit-and-master-installation/41312/9 "2016-07-05T09:01:35Z")
Hey Christian,
sorry for the late reply.
I was tryining to collect different types of data including syslog and netflow.
I was tryining to have a satelite instance on the branch offices.
Is there any guide or other resource to use to Setup my Environment?
Thanks in advance.
Best Regards
Daniel
---
### Author:  [@DanMa](https://discuss.elastic.co/u/DanMa)
#### Post date: [August 2, 2016, 8:02am UTC](https://discuss.elastic.co/t/elk-satelit-and-master-installation/41312/10 "2016-08-02T08:02:24Z")
Can no one help me with this issue?
---
### Author:  [@DanMa](https://discuss.elastic.co/u/DanMa)
#### Post date: [June 11, 2017, 12:42pm UTC](https://discuss.elastic.co/t/elk-satelit-and-master-installation/41312/11 "2017-06-11T12:42:20Z")
Has nobody a solution for me?
I need as Christian right said redis only as a buffer.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 11, 2017, 2:23pm UTC](https://discuss.elastic.co/t/elk-satelit-and-master-installation/41312/12 "2017-06-11T14:23:08Z")
The latest version of Logstash has support for [persistent queues (PQ)](https://www.elastic.co/guide/en/logstash/5.4/persistent-queues.html), allowing received data to be written to disk, which can be used as a buffer. You should therefore now be able to deploy it as follows:
Filebeat ---\> Logstash (with PQ) ---(VPN tunnel)---\> Logstash (with PQ) ---\> Elasticsearch
The first Logstash instance can also collect data that is not file-based through other input plugins. The Logstash to Logstash link supports encryption as well as compression.
---
### Author:  [@DanMa](https://discuss.elastic.co/u/DanMa)
#### Post date: [June 12, 2017, 11:39am UTC](https://discuss.elastic.co/t/elk-satelit-and-master-installation/41312/13 "2017-06-12T11:39:48Z")
Hey Christian, thank you for your quick reply.
The Site where you have planned the filebeat should be our Remote Location.
The Site where you have planned the elasticsearch should be our Central Location.
All Logs (Windows Logs, Firewall Logs, Switch Logs...) on the Remote Location should be collected from one Collector, i think it should be Logstash, because Filebeat has no input for logs over the port 514.
So is the filebeat really needed on the Remote Site? On the Remote Site should be all applications (Logstash, Filebeat?, ..) on one Server. For the Central Site is for every application one Server planned. And we have planned with 2 elasticsearch Servers because of the amount of logs.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 12, 2017, 11:49am UTC](https://discuss.elastic.co/t/elk-satelit-and-master-installation/41312/14 "2017-06-12T11:49:32Z")
If you have no logs that are file-based you may not need to use Filebeat. I would recommend using 3 Elasticsearch nodes if you are looking for resilience and high availability.
---
### Author:  [@DanMa](https://discuss.elastic.co/u/DanMa)
#### Post date: [June 21, 2017, 12:00pm UTC](https://discuss.elastic.co/t/elk-satelit-and-master-installation/41312/15 "2017-06-21T12:00:51Z")
Hey Christian,
many thanks for your help.
I built up the solution as you described.
Best Regards
Daniel
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 4, 2022, 4:14am UTC](https://discuss.elastic.co/t/elk-satelit-and-master-installation/41312/16 "2022-11-04T04:14:16Z")