Filebeat Dynamic Output File Path Configuration Issue

Elastic Stack Beats
1

I'm trying to configure Filebeat to dynamically output files to paths and filenames that map to the original source paths and filenames. For example:
Source path to collect: /root/filebeat/log/source_log/*.log
Desired output path: /root/filebeat/log_files/source_log/, with the filename remaining the same as the original.
The idea is to retrieve the path from the event, process it, add a custom field to the environment, and then reference this variable in output.file. However, during testing, the variable values are not being read correctly. The output directly uses the variable name as a string, and logs indicate that the output filename is initialized ​​before​​ data collection begins.
I'd like to ask: Does Filebeat support dynamic configuration of output paths? If so, how should it be configured correctly?
Here is my test configuration:
＜

filebeat.inputs:

- type: filestream
  enabled: true
  id: app-logs
  paths: \["/root/filebeat/log/source_log/\*.log"\]

processors:

- script:
  lang: javascript
  id: path_rewriter
  source: |
  function process(event) {
  // 1. Get the complete path of the original log file
  var originalLogPath = event.Get("log.file.path");
  if (!originalLogPath) {
  return; // Skip processing if no path information
  }

    var pathArray = originalLogPath.split('/');
  
    // Target the position before the second-to-last element in the path array
    var insertIndex = pathArray.length - 2;
    // Use splice to insert the new directory 'log_files' before that position
    pathArray.splice(insertIndex, 0, 'log_files');
  
    event.Put("custom_rewritten_file", pathArray[pathArray.length-1]); // Set the filename
  
    var pathArrayWithoutLast = pathArray.slice(0, -1); // Remove the last element (filename)
    var newOutputPath = pathArrayWithoutLast.join('/'); // Rejoin into a path string
    event.Put("custom_rewritten_path", newOutputPath); // Set the custom path
  }
  

logging.level: debug

output.file:
path: '/tmp/filebeat_test/%{\[custom_rewritten_path\]}'  # Try to use the custom path variable
filename: '%{\[custom_rewritten_file\]}'  # Try to use the custom filename variable
permissions: 0644
create_parents: true
codec.format:
string: '%{\[custom_rewritten_file\]} %{\[message\]}'

The main issue is that the output uses the literal string %{[custom_rewritten_path]}and %{[custom_rewritten_file]}instead of their values. Logs suggest the output filename is determined very early, before event processing. Any insights or correct configuration examples would be greatly appreciated.

1000279152
10002791521514×696 84.5 KB

2

I don't think you should escape the [ and ], this is basically saying that the field name contains a literal [ and ].

Try to use just %{[custom_rewritten_path]} and the same for the other fields.

Also, your input can be just this:

- type: filestream
  enabled: true
  id: app-logs
  paths: 
    - "/root/filebeat/log/source_log/*.log"
3

Hi. Your filebeat configuration looks like AI generated code. For example create_parents IS NOT valid config reference to the key output.file. You can view the full config reference here for filebeat: filebeat.reference.yml | Beats

Secondly, filename does not support string interpolation, only path does : beats/libbeat/outputs/fileout/config.go at 4ded660aa9fc59fc919e9d2f54f8d476e570ef04 · elastic/beats · GitHub

And since we already access to path variable, it's just matter of extracting the filename. We can create a directory to be the filename and store a generic filename in the directory. This is a workaround.

For example, if the log file is "prod.log" it will give the following path as output:
`/tmp/filebeat_test/prod.log/output.log'

Following config is untested and may or may not work!

filebeat.inputs:
- type: filestream
  enabled: true
  id: app-logs
  paths:
    - /root/filebeat/log/source_log/*.log
  processors:
    - dissect:
        tokenizer: "%{}/%{original_file_name}"
        field: "log.file.path"
        target_prefix: ""

output.file:
  path: "/tmp/filebeat_test/%{[original_file_name]}"
  filename: 'output.log'
  permissions: 0644
  codec.format:
    string: "%{[original_file_name]} %{[message]}"

And word of advice, don't use AI. It's not a magic black 8-ball that give the solution to your problems. It's not a search engine, it's not a crystal ball. It's just a fancy autocomplete tool that will insert what ever it sees fit best.

1 Like