Send with many filebeat to one beat. "Inputbeat"

Elastic Stack Beats
,
1

Hello,
situation:
We use Logstash as a forwarder. Many filebeats send to one Logstash. (filebeat-->forwarder-logstash-->logstash-->elastic)

But the logstash is only a forwarder and very unstable.

Now my question:
Is it possible to set a beat with input beats and output logstash exists?
(filebeat-->beat->logstash-->Elasticsearch)

I think about a "Inputbeat" with output to Logstash.

2

Can you show the configuration files of your current situation to explain in more detail what the problem is and what you are trying to achieve?

3

i want to erase the forward-logstash becouse the unstable/"to-big"/sometimes buggy and i think some lightweight beats is better.

(filebeat-->forwarder-logstash-->logstash-->elastic) now

(filebeat-->forwarder-beats-->logstash-->elastic) wish

forwarder-logstash
Config:

input {                                                                               
  beats {                   
    port  => XXXX                                                                    
    tags  => ["filebeat"]    
    ssl => true
    ssl_certificate_authorities => ["/path/to"]
    ssl_certificate => "/path/to"
    ssl_key => "/path/to"
    ssl_verify_mode => "force_peer"
    client_inactivity_timeout => 600
    tls_min_version => 1.2
   }

output {
  lumberjack {
     codec           => json
     hosts           => ["XXXXX"]
     port            => XXXX
     ssl_certificate => "/path/to"
   }
4

Not sure what you are trying to solve.

@leandrojmp has good advice on using multiple pipelines

and then there is official documentation on pipeline to pipeline communication.

Trying to understand why you need your current set-up, the reasoning behind that setup and why it was built that way.

5

It is not possible, beats only has outputs to Elasticsearch, logstash, kafka, redis, file and console.

I'm trying to understand your scenario, why the need for the forwarder.

You have multiple beats in different machines that are allowed to send data only to this logstash forwarder and then this logstash forwarder has access to the other logstash machine?

Also, working as a forwarder is a simple thing for logstash to do, if it is not working well and seems unstable, maybe there is some underlying issue causing this. How did arrived at the conclusion that it is logstash that is unstable? What do you have in the logs?

What are the versions of filebeat and logstash?

6

For some special network-reasons we need a one-one connection. So we used this:

But we are not happy with the output plugin.

We have still the error message: Improve "Whoa we shouldn't get this frame" message · Issue #22 · logstash-plugins/logstash-output-lumberjack · GitHub

Version: 7.14 Logstash

7

Yeah, that is what I thought could be the reason.

There is not much to do as Filebeat has limited output options.

You could try to remove the logstash forwarder and use a proxy instead, for example HAProxy as a SSL Passthrough with tcp mode.

Or you could use Kafka as a message broker in some place.

8

Thanks leandorjmp - Or we develop a forward-beats :wink:

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.