Filebeat send logs to different port

Elastic Stack Beats
1

Hi there,

i have a question according to the title of this topic. if i have 8 logstash, then i config the filebeat to send the logs to 4 logstash using port 5045 and 5090 to another 4 logstash.

is it possible? if yes, what are the drawbacks?

I've tried the situation above but only use 2 logstashes. it's work but after that, i rollback it again to using only one port because I don't know what will happen if I keep using that config.

Thanks

2

Yes, this is possible, the behaviour is explained in the documentation.

If you use true in the loadbalance option, Filebeat will load balance the requests between the Logstash hosts configured in the hosts option.

If you use false, which is the default, filebeat will randomly choose one Logstash host to send the events, if this hosts becomes unreachable it will randomly choose another Logstash host.

3

so if i make a config based on the situation i explain above, with loadbalance equal true it won't be a problem?

hosts: ["10.11.12.13:5045","10.11.12.14:5045","10.11.12.15:5045","10.11.12.16:5045","10.11.12.17:5090","10.11.12.13:5090","10.11.12.13:5090","10.11.12.13:5090"]

is it safe if i configure like that? i'm afraid if i configure like that there will be misunderstanding. i mean, the logs that should be send using port 5045, due to loadbalance is true, it will be send to port 5090 too. which is not true

4

Oh I see, This is a different thing, it is not about sending logs to different ports, it is about sending different kinds of logs to different ports.

Filebeat will make no distinction to where send the logs, it will loadbalance between the configured servers.

If you are running different pipelines on por 5045 and port 5090 and want to send some logs to just port 5045 and some logs to just port 5090, then this is not possible with filebeat.

Filebeat can have only one output, what you want is to have two different outputs.

5

so the solution is having 2 filebeat or i can configure like this in 1 filebeat?

output.logstash:
  hosts: ["localhost:5045", "localhost:5045"]
  loadbalance: true

output.logstash:
  hosts: ["localhost:5090", "localhost:5090"]
  loadbalance: true
6

oh sorry, i think there's no another way except having 2 filebeat right? @leandrojmp

7

You can do this in a single instance of Filebeat as long as you are tagging the events and then using a when clause. Filebeat conditional output logstash has an example of that.

1 Like
8

image
image1315×118 8.11 KB

i think it's not working with filebeat 7.17.0 i see the logs and it shows me that filebeat just send to port 5049 while in config i was define 5044 too but both output has same IP. the difference is just the port

i followed exactly like an example you gave before. but this is already modified, i want to try another way
image

and the logs still show me the same thing

image
image1313×90 6.21 KB

9

I don't think this works as it would configure multiple outputs and filebeat does not support it.

For what I know the conditional in filebeat output does not work at the hosts level, for example check this similar forum post for the elasticsearch output.

Also check this github issue about why filebeat does not support multiple outputs.

In your case you have two different approaches.

The first one is easier, send everything to the same Logstash port and on this port create a logstash pipeline that will redirect it to other pipelines according to some conditional, this is called the distributor pattern on pipeline-to-pipeline communication as explained in the documentation.

Another option would be to use Kafka and send the data to two different topics, this can be done conditionally in filebeat since the output would be the same, just the topic would be change (there are a couple of questions about this in the forum).

10

Please don't post pictures of text, logs or code. They are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them :slight_smile:

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.