Send some extra fields with filebeat apache2 and system module

[cleverrocks (Nasir Ahmad)]

I am using the filebeat modules apache2 and system for logging. Below is my configuration.

filebeat.modules:
- module: system
  syslog:
    enabled: true
    var.paths: ["path/syslog"]
  auth:
    enabled: true
    var.paths: ["path/auth.log"]

- module: apache2
  access:
     enabled: true
     var.paths: ["path/access.log"]
  error:
     enabled: true
     var.paths: ["path/error.log"]

I want to ship the following fields along each log,

  fields:
    log_type: access
    APP:  app_name
    HOSTNAME: host_name
    CUSTOMER: some_customer
    ROLE: some_role
    STAGE: some_stage

How can i send these fields with the above mentioned 4 log files using modules?

I am also using filebeat.prospectors and the above fields are working fine with the prospector but i need to send with modules logs as well.

Thank you

[steffens (Steffen Siering)]

Checking the apache2 configuration template, I don't think it's possible right now. Filebeat modules are configuration-only objects. The configuration template is defined in apach2/access/config/access.yml. You can see the paths variable (configured via var.paths in filebeat.yml) being expanded in the template. You should be able to modify/fork the module definition to allow for custom fields/settings to be set in a similar fashion.

[cleverrocks (Nasir Ahmad)]

Thank you @steffens. I read the module in detail and came to the point that i need expert suggestion on this because i was also able to find the var.paths in filebeat.yml file but i was looking for a way to inject these fields with each log that's why i need your advice on this.

Regards.

[willemdh]

Nasir,

To start, you would ideally create a new template for filebeat which overrides the default template (higher order). That way you won't need to create a new template manually each Beats update. Then you can add extra fields with input, for example:

- module: apache2
  access:
    enabled: true
    var.paths: ["/var/log/httpd/access_log","/var/log/httpd/ssl_access_log","/var/log/httpd/ssl_request_log"]
    input:
      fields_under_root: true
      fields.dig.app.name: "Nagios XI"
      fields.dig.app.type: "Apache Access Logs"
      fields.dig.source: "filebeat"
      fields.dig.domein: "ses"
  error:
    enabled: true
    var.paths: ["/var/log/httpd/error_log","/var/log/httpd/ssl_error_log"]
    input:
      fields_under_root: true
      fields.dig.app.name: "Nagios XI"
      fields.dig.app.type: "Apache Error Logs"
      fields.dig.source: "filebeat"
      fields.dig.domein: "ses"

The only question you still need to answer is how you populate these fields.. Maybe if each customer has a separate logfile and if you can specify the same module multiple times? (didn't test that)

Grtz

Willem

[cleverrocks (Nasir Ahmad)]

Hi @willemdh,

I am populating these fields in an filebeat template. I have parameterized it and installing this agent in each customer's instance pointing to a centeral ELK server. I am using Puppet for this automation.

Thank you for the idea, i will implement that and keep this post updated if it works,

Best,
Nasir