Collect logs from multiple machine, what needs to be installed?

[Jay_Timbadia]

Continuing the discussion from How to collect the logs from multiple machines to my server efficiently?:

Hi @jsoriano, really followed the chat. Just one thing, I have logs placed in different machine, so should I install beats there and configure it.
I have installed ELK (all three) in different machine.

So the way I should proceed is, I should install ONLY beats in machine which has logs, and the machine where ELK is there will take care of everything right?

So only beats installation to log machine is required, RIGHT?

Thank you!
Jay

[jsoriano (Jaime Soriano)]

Hey @Jay_Timbadia.

Yest, you only need to install Filebeat on the machines where you want to collect logs from. In the machines where you are running the Elastic Stack you can also install Beats if you want to collect logs or metrics from there.

You can also use elastic-agent now to collect logs.

[Jay_Timbadia]

Thank you @jsoriano for quick reply.

I am also facing one more issue.
I have logstash running at 5044 on ELK machine. I have one conf file already existing with TCP input, now I want to create one more conf file for beats input.

So should I make different conf file and place both in conf.d and restart logstash, or how?

Since I tried above approach, it was not working.

Can u pls help?

Jay

[Jay_Timbadia]

Also pls see this chat, Receive multiple Logstash inputs with TCP, UDP and Beats - #3 by sakshat

this person has two logstash config files, but different logstash port number.

So in order to run two different config files, do we need to logstash instance?
Or we can run two config files with same port number?

Will below one work?

input {
	beats {
    port => 5044
    type => 3dlog
  }
}

filter {}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "logs-%{+yyyy.MM.dd}"
  }
}

input {
  tcp {
    port => 5044
    type => syslog
  }
}

filter {
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "app-logs-%{+yyyy.MM.dd}"
  }
}

[jsoriano (Jaime Soriano)]

Yes, it should be possible to use multiple configuration files, you will need to add them to the conf.d directory of your logstash configuration directory, and it seems they need a .conf extension. You would need to restart Logstash after modifying the configuration.

Take a look to the documentation about configuration files: Logstash Configuration Files | Logstash Reference [8.7] | Elastic

[leandrojmp (Leandro Pereira)]

@Jay_Timbadia

So in order to run two different config files, do we need to logstash instance?

You can run multiple independent configurations on the same logstash instance, just need to configure pipelines.yml to point to the configuration files, check the documentation for more information on this.

@Jay_Timbadia

Or we can run two config files with same port number?

You can't use the same port in different configurations, you will need to use different ports, if you are using port 5044 on tcp input, you cannot use this port on the beats input.