Accessing log files on a remote Unix server from a Windows server

Elastic Stack Logstash
1

---------- BACKGROUND ----------
I am setting up a production cluster that has a mix of Windows and Unix servers. The cluster will involve dozens of servers.

I have one master Windows server that is running a single instance of Logstash, Elasticsearch, and Kibana for the entire cluster. The one Logstash instance uses multiple pipelines to organize the logs from the servers in the cluster.

For the unique requirements of this production cluster, I do not want to run any additional applications (like Filebeat) on any of the servers except for the master node. As I have read in the forum, using Filebeat is the preferred method of collecting logs on remote servers. However, in my case, my remote servers are collecting system performance metrics in order to test software products. It is essential that no external applications interfere with system performance. So, I want to retrieve logs from remote servers without the use of additional forwarding applications.

---------- PROBLEM ----------
First of all, all of the servers in the cluster on the same subnet, so the master node can access the file systems of any machine on the subnet. From my Logstash instance running on the Windows master node, I have figured out how to retrieve logs from remote Windows servers. I can do this by configuring my Logstash config file like so:

input {
file {
path => "//windows-host-name/c$/temp/logs/*"
}
}

This works perfectly, but I am having problems accessing the logs on Unix machines. I have made input blocks similar to the one above, but instead using a Unix host name and omitting the "c$".

input {
file {
path => "//unix-host-name/tmp/logs/*"
}
}

When I start this pipeline, Logstash starts the pipeline successfully, but no events are collected and no documents are created.

Is there a way to access Unix file systems remotely from a Windows server?
If not, given my requirements, what would you recommend I do instead?

Thanks in advance for the help <3

2

It is essential that no external applications interfere with system performance. So, I want to retrieve logs from remote servers without the use of additional forwarding applications.

Retrieving logs from a remote computer system's local storage isn't free. There must be some piece of software on the remote side that serves the contents of the files.

This works perfectly, but I am having problems accessing the logs on Unix machines. I have made input blocks similar to the one above, but instead using a Unix host name and omitting the "c$".

That only works if you install Samba on the hosts.

3

Thank you, Magnus, for your reply! I am looking into Samba at the moment. I will update on my progress.

4

The standard way of getting logs from a remote machine is by using an agent such as Logstash. Sharing the logs via SMB/CIFS works but is not recommended.

5

May I ask why it is not recommended?
(If I was to install Samba on all my Unix servers and then access remote logs from within the Logstash configuration file)

I have decided to start by using Filebeat as a log shipper on each of my servers. I'll perform some tests on Filebeat's memory utilization before I attempt to use Samba as a solution. Samba / SMB is not out of the question, though.

6

May I ask why it is not recommended?

  • The setup is more complicated.
  • The attack surface is greater since it relies on one or more open ports on the server in question.
  • It's unclear what corner cases you can run into when tracking a log file accessible via SMB/CIFS. Logstash relies on inode and device numbers and I don't know if they can be accessed in that case.

Sharing the logs via NFS or another network file system is of course also possible but still not recommended.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.