How to forward logs from a logstash “agent” to “master”

Elastic Stack Beats
1

Until now my ELK setup looks like this

Logstash Forwarder -> Logstash 1.4 -> Elstatic Search

Now I want to replace the Forwarder by a full blown Logstash, so I
can collect data from more inputs than just files and can get rid of the
lumberjack protocol with its need to secure communication (which made
always trouble). I also switched to Logstash 2.0.0 and have now the
following setup

Logstash 2.0.0 -> Logstash 2.0.0 -> Elastic Search

I hoped I could use not secured filebeat protocol between the first
logstash instance and the second, but I don't see any (File)beat output
for logstash. For now I'm using http, but this seems to be far to slow.

Is there a way to use Filebeat between 2 full blown logstash
instances, or what would be a appropriate way to let them communicate
without the need to install more infrastructure like messaging providers
or things like that?

2

Have you tried using the TCP Input and TCP Output plugins?

3

The idea behind Filebeat is that it's a lightweight shipper that you install as an agent on your servers to collect data from all your servers and send it to a central Logstash to enrich, parse and transport data. So, the flow would be: Filebeat -> Logstash 2.1.0 -> Elasticsearch.

Excepting files, what data would you like to collect? Maybe other Beat can help you here :slight_smile:

Edit: To answer your question, Logstash has a beats-input plugin, but now a beats-output plugin yet. It might get one in the future. Until then, you could use tcp input & tcp output.

4

TCP output / input should work. As an alternative to have encryption and compression you can use lumberjack input / output: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-lumberjack.html

5

At least in future I will need to connect to JMX beans.
Since we are monitoring large java server applications small footprint is not a requirement, there is enough memory in thats servers. Flexibility is the more important requirement.

6

TCP, reasonable! Could have come myself to that :grinning:

7

@Michael_Bulla Based on this I also created the following issue which could be interesting in the future: https://github.com/elastic/logstash/issues/4282