Override filebeat input paths using command line configuration override?

[tolland (Tom)]

I have a filebeat filestream input which parses a complicated message format. There are a few edge cases I'd like to create standalone tests for. I'd like to test single instances of the the message format from the command line, so basically pass the input.d config file to filebeat but override the paths parameter to point at my sample log, rather than the live log file.

I have a sample line of my format in a file like so /path/to/sample1.log

2023‐07‐27T22:10:55Z,field1,field2,some,other,stuff,{"key":"val"},more,stuff

and a input configuration in inputs.d/my-filestream-id.yml

- type: filestream
  id: my-filestream-id 
  paths:
    - /var/log/my-complicated-logfile.log
  ...
  processors:
     xxxx
  parsers:
     yyyy

so basically I want to do something like this from the command line:

filebeat -E filebeat.config.inputs.path=inputs.d/my-filestream-id.yml \
    -E 'filebeat.inputs.my-filestream-id.paths=["/path/to/sample1.log"] \
    -E output.console.enabled=true \
    -e --once

so it spits out the processed message to the console.

Is this possible to do something like this?

Edit:

I tried a few variations like:

filebeat -E filebeat.config.inputs.path=inputs.d/my-filestream-id.yml \
    -E 'filebeat.inputs.0.paths=["/path/to/sample1.log"] \
    -E output.console.enabled=true \
    -e 

These don't cause the overridden path to be used, still is polling the original path in the the inputs.d config file

Many Thanks

[jsoriano (Jaime Soriano)]

Hey @tolland,

I am not sure if this is possible. I think configuration overrides only work on the main configuration file. You would need to put your inputs configuration in the main file.

You can try though to use environment variables, you could set a default in your configuration file, and override it when running filebeat.
Find here docs about using environment variables in configuration: Use environment variables in the configuration | Filebeat Reference [8.9] | Elastic

Btw, if you have log files that are complicated to parse, you can also consider using Ingest Pipelines, that offer many more options and have a Simulate API for testing.

[tolland (Tom)]

Yes the env thing will work! not sure how I missed that. Many Thanks!

Also, I definitely would have used ingest pipelines if I was starting from scratch.