Pull logs via ssh - push not possible

Elastic Stack Logstash
1

We have some servers in remote data centers where we can open a ssh from our central server to the remote-server. But the remote-server (where the logs get created) can't push the logs to our central server.

What is the best way to get the logs to the central server?

I would like to avoid that the process who pulls the logs reads regular log files in text format and remembers inodes.

This leads to dropped records sooner or later. Example logrotate on the remote host truncates the log files ...

How to provide the logs on the remote-host, so that fetching them via ssh works flawlessly?
And how to pull them?

Please be patient, I am new to LogStash :slight_smile:

Both sides (remote host and central server) run linux and we have full control over them.

2

Have you considered doing a pipe input from a shell script something like this?

#!/bin/bash
while true ; do
    ssh -i sshKeyFile  user@host   "tail --lines=100 --follow=name --retry  /some/path/to/logfile.log"
    sleep 3
end

a tail --follow=name --retry (short version: tail -F ) will continue to follow the logfile name, even when the file is rotated.

The --lines=100 is a swag at the maximum number of lines (and therefore events) you'd miss in the "sleep 3" at the end of the script + time spent rigging a new ssh session. Things like this are why it's important to use the timedate from the logfile entry instead of (or instead of only) the timedate it came in to logstash.

The while loop is so the process survives possible death of the ssh session.

The sleep is to keep the script from spinning out of control if, for some reason, the ssh totally fails, such as the key file gets deleted, the remote host is unreachable, etc.

3

Yes, you solution on tail could work.

But I am a wimp and scaredy-cat. I don't want to loose a single line.

If the network between remote-host and central server is down for some minutes, then logs get lost.

I am afraid - loosing logs must not happen.

4

Is there really not store-and-forward solution for logstash?

5

Hi,

I know this is an old thread, but i still haven't seen an elegant solution for an agent-less Logstash pull from multiple *nix servers. I'm leaning toward using sshfs + a file input, but a more generic SSH based input option would be better.