Copying Files to Central Remote Server Using Beats/LogStash

[arpitwanchoo (arpit)]

Hi

I am trying to design the centralized logging structure for our org.

I was looking for a following pipeline :
RemoteServers(FileBeat) => Central Server (LogStash) => ElasticSearch

I want FileBeat to copy files from client servers to a central repository (specific directory) and then Logstash to read from copied files instead of directly pushing data to logstash from FileBeat

I am trying to do this because I want all my log files in a central location and only filtered content will go to elastic search. There may be situations when I will use "grep" command in files on central server (where all log files are present)

So can this role of "scp" be configured by fileBeat ?

[warkolm (Mark Walkom)]

Not directly.
You'd need FB>LS>file.

[arpitwanchoo (arpit)]

@arpitwanchoo

I am trying to do this because I want all my log files in a central location and only filtered content will go to Elasticsearch. There may be situations when I will use "grep" command in files on central server (where all log files are present)

So can it be that the central server is having to log stash running. The first one accepting data from file beats and creating files on disk. Wile the other one reading files from disk and pushing filtered data to Elasticsearch.

Or Is it possible to configure single log stash in a way that it accepts data from FB and writes two streams (one to disk and one to ES) ?

[warkolm (Mark Walkom)]

@arpitwanchoo

Or Is it possible to configure single log stash in a way that it accepts data from FB and writes two streams (one to disk and one to ES) ?

Yes, you can do that.