# How to collect logs from "outside" Hosts?

**URL:** https://discuss.elastic.co/t/how-to-collect-logs-from-outside-hosts/146325
**Category:** Logstash
**Created:** [August 28, 2018, 11:14am UTC](https://discuss.elastic.co/t/how-to-collect-logs-from-outside-hosts/146325 "2018-08-28T11:14:59Z")
**Posts on this page:** 6
**Page:** 1

<div class="post-metadata">

### Author: ![MarcusCaepio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcuscaepio/32/32458_2.png) [@MarcusCaepio](https://discuss.elastic.co/u/MarcusCaepio)
#### Post date: [August 28, 2018, 11:14am UTC](https://discuss.elastic.co/t/how-to-collect-logs-from-outside-hosts/146325/1 "2018-08-28T11:14:59Z")

</div>

Hi all,  
Currently I am building a Logging Infrastructure based on ELK.  
Status Quo I am collecting network logs via Syslog + Logstash Input, Server Logs via Filebeat + Logstash Input (to have the dynamic for custom prospectors) or Winlogbeat.

As long as this is internal, there is no problem.  
But what is the best practice for server, which are standing "outside"? A push from the server to the indexer will not be allowed. The indexer have to pull the data anyhow.  
The network devices, for example, only can do the syslog protocol (514). Because of standardization the Server should use the beats, too. X-Pack Security is not available.

Any suggestions about best practices are very welcomed.  
Thanks in advance!

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [August 28, 2018, 2:15pm UTC](https://discuss.elastic.co/t/how-to-collect-logs-from-outside-hosts/146325/2 "2018-08-28T14:15:43Z")

</div>

Can you place a broker (Redis, RabbitMQ, Kafka, ...) in between that both parties can connect to?

---

<div class="post-metadata">

### Author: ![MarcusCaepio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcuscaepio/32/32458_2.png) [@MarcusCaepio](https://discuss.elastic.co/u/MarcusCaepio)
#### Post date: [August 28, 2018, 2:33pm UTC](https://discuss.elastic.co/t/how-to-collect-logs-from-outside-hosts/146325/3 "2018-08-28T14:33:52Z")

</div>

Yes, that was my first thought.

Filebeat (outside) -\> Redis/Rabbit/Kafka (outside) \<- Logstash (Inside) -\> Elastic (Inside).  
But what to choose? Redis? Rabbit? Kafka? I don't have any experience with all of them till now 😃

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [August 28, 2018, 6:27pm UTC](https://discuss.elastic.co/t/how-to-collect-logs-from-outside-hosts/146325/4 "2018-08-28T18:27:44Z")

</div>

Well, any of them will do the job. It's really up to you.

---

<div class="post-metadata">

### Author: ![MarcusCaepio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcuscaepio/32/32458_2.png) [@MarcusCaepio](https://discuss.elastic.co/u/MarcusCaepio)
#### Post date: [September 10, 2018, 8:06am UTC](https://discuss.elastic.co/t/how-to-collect-logs-from-outside-hosts/146325/5 "2018-09-10T08:06:52Z")

</div>

Are there no best practices? Or no recommendations of elastic, what works best?  
I will have to handle many many messages. We are talking about several thousands udp messages per second. I heart that some broker crash at a certain amount of messages.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 8, 2018, 8:07am UTC](https://discuss.elastic.co/t/how-to-collect-logs-from-outside-hosts/146325/6 "2018-10-08T08:07:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
