Copying Files to Central Remote Server Using Beats/LogStash

Elastic Stack Logstash
1

Hi

I am trying to design the centralized logging structure for our org.

I was looking for a following pipeline :
RemoteServers(FileBeat) => Central Server (LogStash) => ElasticSearch

I want FileBeat to copy files from client servers to a central repository (specific directory) and then Logstash to read from copied files instead of directly pushing data to logstash from FileBeat

I am trying to do this because I want all my log files in a central location and only filtered content will go to elastic search. There may be situations when I will use "grep" command in files on central server (where all log files are present)

So can this role of "scp" be configured by fileBeat ?

2

Not directly.
You'd need FB>LS>file.

3

So can it be that the central server is having to log stash running. The first one accepting data from file beats and creating files on disk. Wile the other one reading files from disk and pushing filtered data to Elasticsearch.

Or Is it possible to configure single log stash in a way that it accepts data from FB and writes two streams (one to disk and one to ES) ?

4

Yes, you can do that.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.