Multiple beats 1 input port with multiple index output

Elastic Stack Logstash
1

Hello does anyone knows if is it possible to have multiple beats pointing to the same input port and then having multiple index output. Basically I need to identify which beat arrives and then add a tag to then have a proper output.
Best Regards,
Pedro Cabral

2

I believe you can have multiple beats servers connecting to the same port. I haven't scaled my testing to that yet though.

And yes, you can identify with tags on the beats server. Here is what I do on my filebeats

 filebeat.inputs:
	- type: log
		enabled: true
		paths:
			- /var/log/dmesg
		fields:
			dmesg: true

Notice how I set a field to dmesg: true? That is the check that I do in my pipeline on the logstash server.
Then on my logstash server I have a pipelines.yml that looks like this:

- pipeline.id: filebeats
  config.string: |
    input { 
      beats { 
        port => 5044 
      } 
    }
    output {
      if "process" in  [fields] {
          pipeline {
            send_to => process
          }
      }
      else if "dmesg" in [fields] {
        pipeline {
          send_to => dmesg_log
        }
      }
     }
- pipeline.id: process
  path.config: "/etc/logstash/conf.d/process/*"
- pipeline.id: dmesg
  path.config: "/etc/logstash/conf.d/dmesg/*"

Then inside my /etc/logstash/conf.d/dmesg/dmesg.input.conf

input {
  pipeline {
    address => dmesg_log
  }

Please note the use of the virtual addresses that were used for the dmesg. I called the virtual address "dmesg-log" in the pipelines.yml and then I used the same virtual address as the input for my dmesg.input.conf file. This is how you connect them together.

So the process would be:

  • Assign a unique field in your filebeat.yml for each log file
  • Check for that field inside your pipelines.yml
  • Assign a virtual address if that field is there
  • Add the virtual address for the pipeline.input.conf to the input
3

But this case you are just using filebeats.
I want to use auditbeat, winlogbeat and packetbeat.

4

I haven't done the the field thing with winlogbeat, auditbeat, or patcketbeat. But I believe you can do that.
I do have a winlogbeat sending files to a logstash server on port 5044, and I have different server that is sending filebeat files to the same logstash server on port 5044.

I am not sure if you can send winlogbeat and filebeat from the same server on the same port at the same time. I haven't tested this as of yet.

5

Actually, I just tested this, and I have winlogbeat and filebeat on the same system sending data to logstash at the same time on the same port.

So now I have the following going to a single logstash server on a single port:
Windows 10 sending filebeat logs, and winlogbeat event logs.
Linux server sending filebeat logs.

So yes, you can run multiple beats on the client, sending to the same port on the same server.

6

How can you identify on the logstash if is filebeat or winlogbeat to send with different index to elastic?

7

You can create the fields, like I had posted above. Passing the fields in, it will go all the way through logstash. You can either add the index on the input, output or the filter itself based off of the logic from the fields.

I haven't tested the fields with winlogbeat yet, and my test environment is a little tied up right now. But when it's done processing this massive log file, I will test.

8

It works thank you!

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.