I'm trying to configure Filebeat to dynamically output files to paths and filenames that map to the original source paths and filenames. For example:
Source path to collect: /root/filebeat/log/source_log/*.log
Desired output path: /root/filebeat/log_files/source_log/, with the filename remaining the same as the original.
The idea is to retrieve the path from the event, process it, add a custom field to the environment, and then reference this variable in output.file. However, during testing, the variable values are not being read correctly. The output directly uses the variable name as a string, and logs indicate that the output filename is initialized before data collection begins.
I'd like to ask: Does Filebeat support dynamic configuration of output paths? If so, how should it be configured correctly?
Here is my test configuration:
＜
filebeat.inputs:
- type: filestream
enabled: true
id: app-logs
paths: \["/root/filebeat/log/source_log/\*.log"\]
processors:
- script:
lang: javascript
id: path_rewriter
source: |
function process(event) {
// 1. Get the complete path of the original log file
var originalLogPath = event.Get("log.file.path");
if (!originalLogPath) {
return; // Skip processing if no path information
}
var pathArray = originalLogPath.split('/');
// Target the position before the second-to-last element in the path array
var insertIndex = pathArray.length - 2;
// Use splice to insert the new directory 'log_files' before that position
pathArray.splice(insertIndex, 0, 'log_files');
event.Put("custom_rewritten_file", pathArray[pathArray.length-1]); // Set the filename
var pathArrayWithoutLast = pathArray.slice(0, -1); // Remove the last element (filename)
var newOutputPath = pathArrayWithoutLast.join('/'); // Rejoin into a path string
event.Put("custom_rewritten_path", newOutputPath); // Set the custom path
}
logging.level: debug
output.file:
path: '/tmp/filebeat_test/%{\[custom_rewritten_path\]}' # Try to use the custom path variable
filename: '%{\[custom_rewritten_file\]}' # Try to use the custom filename variable
permissions: 0644
create_parents: true
codec.format:
string: '%{\[custom_rewritten_file\]} %{\[message\]}'
The main issue is that the output uses the literal string %{[custom_rewritten_path]}and %{[custom_rewritten_file]}instead of their values. Logs suggest the output filename is determined very early, before event processing. Any insights or correct configuration examples would be greatly appreciated.
