File Mirroring via FileBeat or Logstash

Elastic Stack Beats
1

I have the following use case for several directories containing log files:

  • Tail these files and write out to another location with the same filename

For example, our app is creating files names myapp_.log and we need to copy the file in near real-time to another folder. rsync or the like won't do it so I was thinking of tailing it using filebeat or logstash and using the file output. I tried the file output in both and got tail several source logs and write out to one destination log but I need to keep the filenames the same and the source -> destination on a one-to-one basis.

2

So you need: like this
/dir1/myapp_1.log -> mirror ->/dir2/myapp_1.log
/dir1/myapp_2.log -> mirror ->/dir2/myapp_2.log
etc.

Not sure how to get the current file name and path as runtime values in FB.
LS is capable for sure.

3

I don't think the path/filename fields in filebeat support any kind of dynamic variables etc. I'll see if I can figure out how to do this in logstash.

4

I have managed with LS with some limitation.

input {
  file {
   path => "/path/temp001.txt"
   start_position => beginning
   sincedb_path => "NUL" # you should have real file /path/sincedb.db
   mode => "tail"
  }
}
filter {

  mutate {
    add_field => { 
        "[@metadata][file]" => "%{[log][file][path]}"
   }
  }

  # get file name
  mutate { gsub => [ "[@metadata][file]", "^.*/", "" ] }

}
output {

    file { 
	codec => line { format =>"%{[message]}" }
	path => "/destpath/%{[@metadata][file]}" 
	flush_interval => 1
	#write_behavior => "append"
	}
}

Limitation:

  • You have to set pipeline.workers: 1 in logstash.yml or pipelines.yml to have lines appended in exact order. If the multithread is used, it will mix lines
  • Cannot use a field at begging of path in output
    *If you use an absolute path you cannot start with a dynamic string. E.g: /%{myfield}/,/test-%{myfield}/are not valid paths*
    I have tried to set the destination field [@metafield][destfile] in the filter, but not supported
  • If you use simple or plain codec in output, you have to remove fields included by default "log", @timestamp, "event"
    The host field is always included in output so should be "", not null, but from some reason for every line is added a space. Default dump to file is: %{host}\s%{message}
output {
file { path => "/path/file.txt"}
}

At the end, it's simple, but not simple :wink:

5

This seems to work directly in logstatsh but I need to test further. It will generate a unique filename for the output based on the filename of the input:

f

ilter {
    grok {
      break_on_match => true
      match => {"[log][file][path]" => "(?<myfilename>[ \w-]+)\.log$"}
     
    }
}

output {
    file {
    path => "C:/logs/DirDest/%{myfilename}.log"
    codec => line { format => "%{message}" }
    }   
}
6

Thanks @Rios ! I'm a noob at ELK so wondering why you used mutate instead of grok? I seemed to get it working with grok but I need to test it under load.

7

Well grok is also regex in background. Why mutate? It's just need a file name not to parse fields.

8

So likely more efficient to use mutate?

9

Just easier.