Filebeat module specific output : why is this a global config?

Elastic Stack Beats
1

I have multiple modules active and like to have each module output saved to a distinct location as I choose file ouput (no elastic integration)

I see the output configuration as part of filebeat.yml, which is a global configuration
output.file:

output.file:
  # Boolean flag to enable or disable the output module.
  enabled: true

  # Configure JSON encoding
  codec.json:
    # Pretty-print JSON event
    pretty: true

  # Path to the directory where to save the generated files. The option is
  # mandatory.
  path: "/var/crystaleye/azure1"

I tried moving the file.output section to module but it does not store any output at all

How to achieve this? I dont find any documentation which covers this as a module configuration
such as part of /etc/filebeat/modules.d/o365.yml

2

It's possible, set everything to make sure that input, processing is working fine.

output.console:
  pretty: true

If you see expected data on console, switch to the file mode. What can be possible:

  • not configured the module, usually path to logs, in your case o365.yml
  • something is not OK in input settings.
  • your data is already read, (re)move the registry file
  • enable debug mode
3

This is the file:

- module: o365
  audit:
    enabled: true

    var.application_id: "xxx-fa26-4a70-8c35-a4be3998df9a"
    retention_period:     60

    var.tenants:
     - id: "xxx-8d5f-4b63-b75c-xxx"
       name: "xxxxx.onmicrosoft.com"
    var.client_secret: "eS18Q~2IUIHUZR0MwITSUsPEMe505SwHDATLXbHC"

    var.content_type:
      - "Audit.AzureActiveDirectory"
      - "Audit.Exchange"
      - "Audit.SharePoint"
      - "Audit.General"
      - "DLP.All"


    poll_interval: 40s

output.file:
  enabled: true

  codec.json:
    pretty: true

  path: "/var/crystaleye/azure1"
4

The module settings look OK. Add log and set debug level for more info.

5

Tried this but dont find any relevant debug logs. This log shows up frequently

{"log.level":"debug","@timestamp":"2023-05-20T10:09:41.546+0530","log.logger":"cfgfile","log.origin":{"file.name":"cfgfile/reload.go","file.line":194},"message":"Scan for new config files","service.name":"filebeat","ecs.version":"1.6.0"}

So I am thinking the file output mode is not often used and this is a basic bug and not reported by the folks?
The question is about filebeat logs and module output configuration should be in diffferent levels

Configuration of

  • filebeat logs : filebeat.yml
  • module specific output: modules.d/<module.yml>
6

I don't call it a bug, since such a requirement is not specified may be ? Can we consider this as a feature request or an improvement ?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.