File Mirroring via FileBeat or Logstash

I have the following use case for several directories containing log files:

  • Tail these files and write out to another location with the same filename

For example, our app is creating files names myapp_.log and we need to copy the file in near real-time to another folder. rsync or the like won't do it so I was thinking of tailing it using filebeat or logstash and using the file output. I tried the file output in both and got tail several source logs and write out to one destination log but I need to keep the filenames the same and the source -> destination on a one-to-one basis.

So you need: like this
/dir1/myapp_1.log -> mirror ->/dir2/myapp_1.log
/dir1/myapp_2.log -> mirror ->/dir2/myapp_2.log
etc.

Not sure how to get the current file name and path as runtime values in FB.
LS is capable for sure.

I don't think the path/filename fields in filebeat support any kind of dynamic variables etc. I'll see if I can figure out how to do this in logstash.

I have managed with LS with some limitation.

input {
  file {
   path => "/path/temp001.txt"
   start_position => beginning
   sincedb_path => "NUL" # you should have real file /path/sincedb.db
   mode => "tail"
  }
}
filter {

  mutate {
    add_field => { 
        "[@metadata][file]" => "%{[log][file][path]}"
   }
  }

  # get file name
  mutate { gsub => [ "[@metadata][file]", "^.*/", "" ] }

}
output {

    file { 
	codec => line { format =>"%{[message]}" }
	path => "/destpath/%{[@metadata][file]}" 
	flush_interval => 1
	#write_behavior => "append"
	}
}

Limitation:

  • You have to set pipeline.workers: 1 in logstash.yml or pipelines.yml to have lines appended in exact order. If the multithread is used, it will mix lines
  • Cannot use a field at begging of path in output
    *If you use an absolute path you cannot start with a dynamic string. E.g: /%{myfield}/,/test-%{myfield}/are not valid paths*
    I have tried to set the destination field [@metafield][destfile] in the filter, but not supported
  • If you use simple or plain codec in output, you have to remove fields included by default "log", @timestamp, "event"
    The host field is always included in output so should be "", not null, but from some reason for every line is added a space. Default dump to file is: %{host}\s%{message}
output {
file { path => "/path/file.txt"}
}

At the end, it's simple, but not simple :wink:

1 Like

This seems to work directly in logstatsh but I need to test further. It will generate a unique filename for the output based on the filename of the input:

f

ilter {
    grok {
      break_on_match => true
      match => {"[log][file][path]" => "(?<myfilename>[ \w-]+)\.log$"}
     
    }
}

output {
    file {
    path => "C:/logs/DirDest/%{myfilename}.log"
    codec => line { format => "%{message}" }
    }   
}

Thanks @Rios ! I'm a noob at ELK so wondering why you used mutate instead of grok? I seemed to get it working with grok but I need to test it under load.

Well grok is also regex in background. Why mutate? It's just need a file name not to parse fields.

So likely more efficient to use mutate?

Just easier.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.