# How does beat communicate with applications and logstash?

**URL:** https://discuss.elastic.co/t/how-does-beat-communicate-with-applications-and-logstash/147392
**Category:** Beats
**Created:** [September 5, 2018, 11:35am UTC](https://discuss.elastic.co/t/how-does-beat-communicate-with-applications-and-logstash/147392 "2018-09-05T11:35:50Z")
**Posts on this page:** 6
**Page:** 1

<div class="post-metadata">

### Author: ![Malay\_Peaas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/malay_peaas/32/35192_2.png) [@Malay\_Peaas](https://discuss.elastic.co/u/Malay_Peaas)
#### Post date: [September 5, 2018, 11:35am UTC](https://discuss.elastic.co/t/how-does-beat-communicate-with-applications-and-logstash/147392/1 "2018-09-05T11:35:50Z")

</div>

Hi,

I am new to Elastic stack, I read about how to configure FileBeat/WinLogBeat etc. to read logs from application/OS and output them to logstash. I would like to know that how does beat communicate with logstash and applications under the hood, what protocol does it use. Also while sending the data to logstash what type of network call it makes, and is this call asynchronous or synchronous?

Thanks and Regards,  
Malay M

---

<div class="post-metadata">

### Author: ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)
#### Post date: [September 5, 2018, 12:09pm UTC](https://discuss.elastic.co/t/how-does-beat-communicate-with-applications-and-logstash/147392/2 "2018-09-05T12:09:33Z")

</div>

Communication with Logstash  
Beats communicate with logstash using the Lumberjack protocol over TCP. You can find the Golang implementation here: [https://github.com/elastic/go-lumber](https://github.com/elastic/go-lumber)  
Beats can forward events both synchronously and asynchronously to LS. This can be configured in the `pipelining` option. By default it's set to two. So events are forwarded asynchronously. On ACK of events a callback provided by Beats is called.

```auto
# Number of batches to be sent asynchronously to Logstash while processing
# new batches.
#pipelining: 2

```

Communication with inputs  
By applications I assume you mean input applications e.g Eventlog. Correct me it it's not what you are interested in.  
Beats use the standard API provided by the applications or external Golang libs. For example in case of Eventlog Windows API is used. For redis `"github.com/garyburd/redigo/redis"` is used.

---

<div class="post-metadata">

### Author: ![Malay\_Peaas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/malay_peaas/32/35192_2.png) [@Malay\_Peaas](https://discuss.elastic.co/u/Malay_Peaas)
#### Post date: [September 5, 2018, 12:26pm UTC](https://discuss.elastic.co/t/how-does-beat-communicate-with-applications-and-logstash/147392/3 "2018-09-05T12:26:42Z")

</div>

Thanks for the details, yes by application I meant input application, also is it possible for me to send logs as events to kafka and then read them using beat? So that I won't have to deal with log file rotation and if so how can I strip off extra kafka logs other than my application logs in beats or do I have to do all this in logstash?

And also I couldn't find latest doc on LumberJack protocol, I found [this](https://github.com/elastic/logstash-forwarder/blob/master/PROTOCOL.md) but it mentions that this doc is deprecated. So is there any latest documentation explaining this protocol?

---

<div class="post-metadata">

### Author: ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)
#### Post date: [September 5, 2018, 1:11pm UTC](https://discuss.elastic.co/t/how-does-beat-communicate-with-applications-and-logstash/147392/4 "2018-09-05T13:11:58Z")

</div>

You can send events to kafka using the kafka output. However, kafka as an input is not supported by Elastic Beats. But I have found one community Beat which might be useful to you: [https://github.com/justsocialapps/kafkabeat](https://github.com/justsocialapps/kafkabeat)

The v1 of the protocol is deprecated. Beats uses v2 and the repo I referred to includes both implementations. Unfortunately, there is no documentation for Lumberjack v2. [https://github.com/elastic/libbeat/issues/279#issuecomment-365496030](https://github.com/elastic/libbeat/issues/279#issuecomment-365496030)

---

<div class="post-metadata">

### Author: ![Malay\_Peaas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/malay_peaas/32/35192_2.png) [@Malay\_Peaas](https://discuss.elastic.co/u/Malay_Peaas)
#### Post date: [September 6, 2018, 5:09am UTC](https://discuss.elastic.co/t/how-does-beat-communicate-with-applications-and-logstash/147392/5 "2018-09-06T05:09:59Z")

</div>

Thanks, this information was helpful enough to get me started on beats and logstash.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 4, 2018, 7:10am UTC](https://discuss.elastic.co/t/how-does-beat-communicate-with-applications-and-logstash/147392/6 "2018-10-04T07:10:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
