How do I add a Custom Field to FileBeat with a Module?

Elastic Stack Beats
1

Below is the top portion of my filebeat yaml.

This configuration works adequately.
However I would like to append additional data to the events in order to better distinguish the source of the logs.
I have gone through all the documentation regarding "field" and "add_fields" and "processors" and "filebeat.inputs:" I cannot seem to get the custom meta data to appear in the logs when I view them in Kibana.
Below is my regular yaml without any custom fields.
I would like to be able to add something like:

 processors:
  - add_fields:
    target: ''
    fields:
      Customer: Customer123
    fields_under_root: true

Or

 filebeat.inputs:
- type: log
  fields:
    Customer: Customer123
  fields_under_root: true

Is it possible to append additional custom fields to logs while also using a module?
If so how would I configure my yaml to do that?

CURRENT YAML

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: XXXXXXXXXXXXX
  namespace: XXXXXXXXXXX
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.modules:
    - module: okta
      system:
        var.url: https://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
        var.api_key: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
      
    output.elasticsearch:
      hosts: ['https://${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      ssl.certificate_authorities:
      - /mnt/elastic/tls.crt`
2

Hey @mstacy,

Something like the configuration for processors you tried should work, but take into account that indentation is important, settings of add_field should have one more indentation level.

Something like this should work:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: XXXXXXXXXXXXX
  namespace: XXXXXXXXXXX
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.modules:
    - module: okta
      system:
        var.url: https://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
        var.api_key: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'

    processors:
    - add_fields:
        target: ''
        fields:
          Customer: Customer123

    output.elasticsearch:
      hosts: ['https://${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      ssl.certificate_authorities:
      - /mnt/elastic/tls.crt`

If this doesn't work, could you check for any error in filebeat logs?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.