# File Mirroring via FileBeat or Logstash
**URL:** https://discuss.elastic.co/t/file-mirroring-via-filebeat-or-logstash/315973
**Category:** Beats
**Tags:** filebeat
**Created:** [October 6, 2022, 12:50pm UTC](https://discuss.elastic.co/t/file-mirroring-via-filebeat-or-logstash/315973 "2022-10-06T12:50:57Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@shocko](https://discuss.elastic.co/u/shocko)
#### Post date: [October 6, 2022, 12:50pm UTC](https://discuss.elastic.co/t/file-mirroring-via-filebeat-or-logstash/315973/1 "2022-10-06T12:50:57Z")
I have the following use case for several directories containing log files:
- Tail these files and write out to another location with the same filename
For example, our app is creating files names myapp\_.log and we need to copy the file in near real-time to another folder. _rsync_ or the like won't do it so I was thinking of tailing it using filebeat or logstash and using the file output. I tried the file output in both and got tail several source logs and write out to one destination log but I need to keep the filenames the same and the source -\> destination on a one-to-one basis.
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [October 6, 2022, 7:56pm UTC](https://discuss.elastic.co/t/file-mirroring-via-filebeat-or-logstash/315973/2 "2022-10-06T19:56:41Z")
So you need: like this
/dir1/myapp\_1.log -\> mirror -\>/dir2/myapp\_1.log
/dir1/myapp\_2.log -\> mirror -\>/dir2/myapp\_2.log
etc.
Not sure how to get the current file name and path as runtime values in FB.
LS is capable for sure.
---
### Author:  [@shocko](https://discuss.elastic.co/u/shocko)
#### Post date: [October 7, 2022, 10:33pm UTC](https://discuss.elastic.co/t/file-mirroring-via-filebeat-or-logstash/315973/3 "2022-10-07T22:33:14Z")
I don't think the [path/filename fields](https://www.elastic.co/guide/en/beats/filebeat/current/file-output.html) in filebeat support any kind of dynamic variables etc. I'll see if I can figure out how to do this in logstash.
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [October 8, 2022, 6:47pm UTC](https://discuss.elastic.co/t/file-mirroring-via-filebeat-or-logstash/315973/4 "2022-10-08T18:47:44Z")
I have managed with LS with some limitation.
```auto
input {
file {
path => "/path/temp001.txt"
start_position => beginning
sincedb_path => "NUL" # you should have real file /path/sincedb.db
mode => "tail"
}
}
filter {
mutate {
add_field => {
"[@metadata][file]" => "%{[log][file][path]}"
}
}
# get file name
mutate { gsub => ["[@metadata][file]", "^.*/", "" ] }
}
output {
file {
codec => line { format =>"%{[message]}" }
path => "/destpath/%{[@metadata][file]}"
flush_interval => 1
#write_behavior => "append"
}
}
```
Limitation:
- You have to set pipeline.workers: 1 in logstash.yml or pipelines.yml to have lines appended in exact order. If the multithread is used, it will mix lines
- Cannot use a field at begging of path in output
`*If you use an absolute path you cannot start with a dynamic string. E.g: `/%{myfield}/`,`/test-%{myfield}/`are not valid paths*`
I have tried to set the destination field [@metafield][destfile] in the filter, but not supported
- If you use simple or plain codec in output, you have to remove fields included by default "log", @timestamp, "event"
The host field is always included in output so should be "", not null, but from some reason for every line is added a space. Default dump to file is: %{host}\s%{message}
```auto
output {
file { path => "/path/file.txt"}
}
```
At the end, it's simple, but not simple 😉
---
### Author:  [@shocko](https://discuss.elastic.co/u/shocko)
#### Post date: [October 9, 2022, 9:53am UTC](https://discuss.elastic.co/t/file-mirroring-via-filebeat-or-logstash/315973/5 "2022-10-09T09:53:09Z")
This seems to work directly in **logstatsh** but I need to test further. It will generate a unique filename for the output based on the filename of the input:
f
```auto
ilter {
grok {
break_on_match => true
match => {"[log][file][path]" => "(?[\w-]+)\.log$"}
}
}
output {
file {
path => "C:/logs/DirDest/%{myfilename}.log"
codec => line { format => "%{message}" }
}
}
```
---
### Author:  [@shocko](https://discuss.elastic.co/u/shocko)
#### Post date: [October 9, 2022, 9:56am UTC](https://discuss.elastic.co/t/file-mirroring-via-filebeat-or-logstash/315973/6 "2022-10-09T09:56:13Z")
Thanks @Rios ! I'm a noob at ELK so wondering why you used **mutate** instead of **grok**? I seemed to get it working with grok but I need to test it under load.
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [October 9, 2022, 10:08am UTC](https://discuss.elastic.co/t/file-mirroring-via-filebeat-or-logstash/315973/7 "2022-10-09T10:08:01Z")
Well grok is also regex in background. Why mutate? It's just need a file name not to parse fields.
---
### Author:  [@shocko](https://discuss.elastic.co/u/shocko)
#### Post date: [October 9, 2022, 8:28pm UTC](https://discuss.elastic.co/t/file-mirroring-via-filebeat-or-logstash/315973/8 "2022-10-09T20:28:43Z")
So likely more efficient to use mutate?
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [October 9, 2022, 9:45pm UTC](https://discuss.elastic.co/t/file-mirroring-via-filebeat-or-logstash/315973/9 "2022-10-09T21:45:48Z")
Just easier.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 6, 2022, 11:46pm UTC](https://discuss.elastic.co/t/file-mirroring-via-filebeat-or-logstash/315973/10 "2022-11-06T23:46:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.