Filebeat module for modsecurity v3

[matthijs42 (Matthijs)]

Hi,

I'm trying to write a new filebeat module for modsecurity v3.

I followed the tutorial on the website (https://www.elastic.co/guide/en/beats/devguide/current/filebeat-modules-devguide.html) but I am stuck on 'create-fields'.

The steps I executed:
Creating the module with:
make create-module MODULE=modsecurity3

I did not adjust any of the generated files at this moment.

Next step:
make create-fileset MODULE=modsecurity3 FILESET=auditlog

The configuration file module/modsecurity3/auditlog/manifest.yml:
module_version: 1.0

var:
  - name: paths
    default:
      - /var/log/modsecurity/**

ingest_pipeline: ingest/pipeline.json
input: config/auditlog.yml

requires.processors:
  - name: user_agent
    plugin: ingest-user-agent
  - name: geoip
    plugin: ingest-geoip

Config file module/modsecurity3/auditlog/config/auditlog.yml:

type: log
paths:
{{ range $i, $path := .paths }}
 - {{$path}}
{{ end }}
exclude_files: [".gz$", "modsec-index", "modsec-index.1"]
json.keys_under_root: true
json.add_error_key: true
close_eof: true

Reasoning for this:
The modsecurity logs are written in json to a log file per event. So there is one json object per file. So immediately close the log file after it has been read.

Config file module/modsecurity3/auditlog/ingest/pipeline.json:

{
  "description": "Pipeline for parsing ModSecurity v3 auditlog logs",
  "processors": [
    {
		"json": {
			"field": "transaction",
			"add_to_root": true
		}
	},{
		"rename": {
			"field": "@timestamp",
			"target_field": "read_timestamp",
			"ignore_missing": true
		}
	},{
		"user_agent": {
			"field": "modsecurity3.auditlog.transaction.request.headers.user-agent"
		}
	},{
		"date": {
			"field": "modsecurity3.auditlog.transaction.time_stamp",
			"target_field": "@timestamp",
			"formats": ["EEE, d MMM yyyy HH:mm:ss z"]
		}
	},{
		"geoip": {
			"field": "modsecurity3.auditlog.transaction.client_ip",
			"ignore_missing": true
		}
	}
  ],
  "on_failure" : [{
    "set" : {
      "field" : "error.message",
      "value" : "{{ _ingest.on_failure_message }}"
    }
  }]
}

At this point the tutorial says to generate a fields.yml by using the following command, but that fails with:
$ make create-fields MODULE=modsecurity3 FILESET=auditlog
Error while generating fields.yml: cannot generate fields.yml: No patterns in pipeline
exit status 2
make: *** [Makefile:28: create-fields] Error 1

I am obviously doing something wrong in the pipeline configuration (pipeline.json). But I cannot find out what I am doing wrong.

Regards, Matthijs