How to use processors in filebeat HAproxy's module?

[aventrax (av3n)]

Hello, I'm new with filebeat and I'm in trouble adding a processor to the haproxy module.

To begin, just adding a tag would be enough, I tried with this config without much luck (Filebeat 7.1.1 on Debian stretch):

/etc/filebeat/modules.d/haproxy.yml

- module: haproxy
  # All logs
  log:
    enabled: true
    # Set which input to use between syslog (default) or file.
    #var.input:
    var.input: "file"

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/haproxy.log"]

    processors:
      - add_tags:
          tags: [test]

The processor is NOT working. If sat globally on /etc/filebeat/filebeat.yml it works.
Any help would be very appreciated. Many thanks.

[Mario_Castro]

Hi @aventrax

Processors can only be used globally on each Filebeat instance (see 3rd paragraph) https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html

[aventrax (av3n)]

Hi @Mario_Castro, I don't understand, reading this under Where are processors valid? , there is something that seems contraddict your sentence: Similarly, for Filebeat modules, you can define processors under the input section of the module definition.

Moreover, here @jsoriano is advising a user that asked a similar question. There is a processor used on apache module, am I wrong?

[Mario_Castro]

Sorry, some parts of the docs must be definitely updated. Anyways, you cannot do maths in a processor (key or value). Supported conditions are https://www.elastic.co/guide/en/beats/filebeat/current/defining-processors.html#conditions

[aventrax (av3n)]

Sorry but my english is not good enough, which document is incorrect? I'm not trying to do very complicated stuff in processor, now I'm trying adding a tag, the final goal is to add a field only if the source.ip field is in a certain network CIDR. To do so I'll use the network condition and I know that I will need to upgrade to (at least) 7.2. Anyway, at the moment I'm only trying to make the processor working in the haproxy module.

[Mario_Castro]

EDIT: Checked, this config should work:

- module: haproxy
  # All logs
  log:
    enabled: true
    # Set which input to use between syslog (default) or file.
    #var.input:
    var.input: "file"

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/haproxy.log"]

    input:
      processors:
        - add_tags:
            tags: [test]

The key was the missing input parent field. The post you mentioned was misleading in this aspect. I have edited already

[aventrax (av3n)]

Many thanks @Mario_Castro , it works! I tried any combination, my first try was very close to your solution but I used "var.input:" in place of "input:" and I've got a duplicated variable error or something like that.

Anyway, thank you!

[aventrax (av3n)]

Just to complete the post, I finally reached my goal. After the ELK upgrade to version 7.5.2, here's the configuration I ended up with.

- module: haproxy
  log:
    enabled: true
    var.input: "file"
    var.paths: ["/var/log/haproxy.log"]
    var.convert_timezone: true

    input:
      processors:

      - decode_csv_fields:
          fields:
            message: decoded.csv
          separator: ":"
          ignore_missing: false
          overwrite_keys: true
          trim_leading_space: true
          fail_on_error: false

      - extract_array:
          field: decoded.csv
          mappings:
            client.ip: 3

      - add_fields:
          when:
            network:
              client.ip: '10.0.9.0/24'
          fields:
            source.site: VPN Network
          target: ''

      - drop_fields:
          fields: ['decoded.csv', 'client.ip']