How does beat communicate with applications and logstash?

Elastic Stack Beats
1

Hi,

I am new to Elastic stack, I read about how to configure FileBeat/WinLogBeat etc. to read logs from application/OS and output them to logstash. I would like to know that how does beat communicate with logstash and applications under the hood, what protocol does it use. Also while sending the data to logstash what type of network call it makes, and is this call asynchronous or synchronous?

Thanks and Regards,
Malay M

1 Like
2

Communication with Logstash
Beats communicate with logstash using the Lumberjack protocol over TCP. You can find the Golang implementation here: https://github.com/elastic/go-lumber
Beats can forward events both synchronously and asynchronously to LS. This can be configured in the pipelining option. By default it's set to two. So events are forwarded asynchronously. On ACK of events a callback provided by Beats is called.

# Number of batches to be sent asynchronously to Logstash while processing
# new batches.
#pipelining: 2

Communication with inputs
By applications I assume you mean input applications e.g Eventlog. Correct me it it's not what you are interested in.
Beats use the standard API provided by the applications or external Golang libs. For example in case of Eventlog Windows API is used. For redis "github.com/garyburd/redigo/redis" is used.

1 Like
3

Thanks for the details, yes by application I meant input application, also is it possible for me to send logs as events to kafka and then read them using beat? So that I won't have to deal with log file rotation and if so how can I strip off extra kafka logs other than my application logs in beats or do I have to do all this in logstash?

And also I couldn't find latest doc on LumberJack protocol, I found this but it mentions that this doc is deprecated. So is there any latest documentation explaining this protocol?

4

You can send events to kafka using the kafka output. However, kafka as an input is not supported by Elastic Beats. But I have found one community Beat which might be useful to you: https://github.com/justsocialapps/kafkabeat

The v1 of the protocol is deprecated. Beats uses v2 and the repo I referred to includes both implementations. Unfortunately, there is no documentation for Lumberjack v2. https://github.com/elastic/libbeat/issues/279#issuecomment-365496030

5

Thanks, this information was helpful enough to get me started on beats and logstash.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.