Filebeat for naxsi waf log

hi, there

I was trying to build a filebeat module for naxsi waf log.

the log is on /usr/local/shadow/logs/shadow_waf.log

the log format is below:

2017/09/07 18:26:21 [error] 1097#0: SHADOW_WAF: 109720&ip=$SQL&score0=8&cscore1=$XSS&score1=16&zone0=URL&id0=1008&var_name0=&content=''

the grok pattern is from logstash config file (which works alright on logstash):

the field.yml file on waf/_meta direcotry:

  • name: waf
    type: group
    description: >
    Contains fields for shadow waf logs.
    • name: severity
      type: keyword
      description: >
      severity of log.
    • name: pid
      type: keyword
      description: >
      process id.
    • name: tag
      type: keyword
      description: >
      The tag of waf log.
    • name: support_id
      type: keyword
      description: >
      support id of the attack.
    • name: client_ip
      type: keyword
      description: >
      client ip address.
    • name: http_host
      type: keyword
      description: >
      app domain.
    • name: http_uri
      type: keyword
      description: >
      access url.
    • name: learning
      type: keyword
      description: >
      the status code of learning.
    • name: version
      type: keyword
      description: >
      version of shadow waf module.
    • name: processed
      type: keyword
      description: >
      number of processed requests.
    • name: total_blocked
      type: keyword
      description: >
      number of total blocked requests.
    • name: blocked
      type: keyword
      description: >
      number of blocked requests.
    • name: type
      type: keyword
      description: >
      type of the web attack.
    • name: score
      type: keyword
      description: >
      the score of the attack.
    • name: zone
      type: keyword
      description: >
      attack part of the http request.
    • name: rule_id
      type: keyword
      description: >
      which id of the rule.
    • name: http_variable
      type: keyword
      description: >
      http variable of the attack.
    • name: attack_content
      type: keyword
      description: >
      actual attack request and body.
    • name: geoip
      type: group
      description: >
      Contains GeoIP information gathered based on the client_ip field.
      Only present if the GeoIP Elasticsearch plugin is available and
      • name: continent_name
        type: keyword
        description: >
        The name of the continent.
      • name: country_iso_code
        type: keyword
        description: >
        Country ISO code.
      • name: location
        type: geo_point
        description: >
        The longitude and latitude.

the waf.yml on waf/config directory:

type: log
{{ range $i, $path := .paths }}

  • {{path}} {{ end }} exclude_files: [".gz"]

the pipeline.json on shadow/ingest directory:

"description": "Pipeline for parsing shadow waf logs.Requires the geoip plugin.",
"processors": [{
"grok": {
"field": "message",
"(?<shadow.waf.time>%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME}) [%{LOGLEVEL:shadow.waf.severity}] %{}#%{NUMBER}: %{WORD:shadow.waf.tag}: %{NUMBER:shadow.waf.support_id}&ip=%{IP:shadow.waf.client_ip}&server=%{IPORHOST:shadow.waf.http_host}&uri=%{GREEDYDATA:shadow.waf.http_uri}&learning=%{WORD:shadow.waf.learning}&vers=%{DATA:shadow.waf.version}&total_processed=%{NUMBER:shadow.waf.processed}&total_blocked=%{NUMBER:shadow.waf.total_blocked}&block=%{NUMBER:shadow.waf.blocked}(&cscore0=%{GREEDYDATA:shadow.waf.type}&score0=%{NUMBER:shadow.waf.score})?(&cscore1=%{GREEDYDATA:shadow.waf.type}&score1=%{NUMBER:shadow.waf.score})?(&cscore2=%{GREEDYDATA:shadow.waf.type}&score2=%{NUMBER:shadow.waf.score})?(&cscore3=%{GREEDYDATA:shadow.waf.type}&score3=%{NUMBER:shadow.waf.score})?(&zone0=%{}&id0=%{NUMBER:shadow.waf.rule_id}&var_name0=%{USERNAME:shadow.waf.http_variable})?(&zone1=%{}&id1=%{NUMBER:shadow.waf.rule_id}&var_name1=%{USERNAME:shadow.waf.http_variable}?)?(&zone2=%{}&id2=%{NUMBER:shadow.waf.rule_id}&var_name2=%{USERNAME:shadow.waf.http_variable}?)?(&zone3=%{}&id3=%{NUMBER:shadow.waf.rule_id}&var_name3=%{USERNAME:shadow.waf.http_variable}?)?(&content='%{GREEDYDATA:shadow.waf.attack_content}')?"
"ignore_missing": true
"field": "message"
}, {
"rename": {
"field": "@timestamp",
"target_field": "read_timestamp"
}, {
"date": {
"field": "shadow.waf.time",
"target_field": "@timestamp",
"formats": ["dd/MMM/YYYY:H:m:s Z"]
}, {
"remove": {
"field": "shadow.waf.time"
}, {
"geoip": {
"field": "shadow.waf.client_ip",
"target_field": "shadow.waf.geoip"
"on_failure" : [{
"set" : {
"field" : "error.message",
"value" : "{{ _ingest.on_failure_message }}"

the manifest.yml on shadow directory

module_version: 1.0


  • name: paths
    • /usr/local/shadow/logs/shadow_waf.log
    • /usr/local/shadow/logs/shadow_waf.log
    • c:/programdata/shadow/logs/shadow_waf.log

ingest_pipeline: ingest/pipeline.json
prospector: config/waf.yml


  • name: geoip
    plugin: ingest-geoip

just follow the developer guide, but when I try to use the module on filebeat.yml, the filebeat failed to start with no logs on /var/log/filebeat/filebeat.

any inputs are appreciated!

Please format your code using the </> button, or markdown style back ticks, it's really hard to read as it is now :slight_smile:

got it ... thanks

I just adjust the style, is it OK?

Are Grok patterns in the pipeline working? You can test it using Simulate API:

Could you please share the log of Filebeat?

the grok patterns are working in my logstash conf.
when I try to start the filebeat with filebeat.yml which call the new module. It just failed to start with no logs on the /var/log/filebeat/filebeat

when I replace the filebeat.yml and restart the filebeat, the log is below:

2017-11-10T22:20:37+08:00 DBG Received sigterm/sigint, stopping
2017-11-10T22:20:37+08:00 INFO Stopping filebeat
2017-11-10T22:20:37+08:00 INFO Stopping Crawler
2017-11-10T22:20:37+08:00 INFO Stopping 1 prospectors
2017-11-10T22:20:37+08:00 INFO Prospector ticker stopped
2017-11-10T22:20:37+08:00 INFO Stopping Prospector: 5287455123581690130
2017-11-10T22:20:37+08:00 INFO Prospector channel stopped because beat is stopping.
2017-11-10T22:20:37+08:00 INFO Crawler stopped
2017-11-10T22:20:37+08:00 INFO Stopping spooler
2017-11-10T22:20:37+08:00 DBG Spooler has stopped
2017-11-10T22:20:37+08:00 DBG Shutting down sync publisher
2017-11-10T22:20:37+08:00 INFO Stopping Registrar
2017-11-10T22:20:37+08:00 INFO Ending Registrar
2017-11-10T22:20:37+08:00 DBG Write registry file: /var/lib/filebeat/registry
2017-11-10T22:20:37+08:00 DBG Registry file updated. 5 states written.
2017-11-10T22:20:37+08:00 INFO Total non-zero values: registrar.states.current=5 registrar.states.update=2 registrar.writes=2
2017-11-10T22:20:37+08:00 INFO Uptime: 8m56.879360288s
2017-11-10T22:20:37+08:00 INFO filebeat stopped.

And here is the filebeat.yml:

###################### Filebeat Configuration Example #########################

  • module: shadow
    enbaled: true
    var.paths: ["/usr/local/shadow/logs/shadow_waf.log"]

#=========================== Filebeat prospectors =============================

#- input_type: log


- /usr/local/shadow/logs/shadow.log

#level: shadow
#fields_under_root: true

#exclude_lines: ["^DBG"]
#include_lines: ["^ERR", "^WARN"]
#exclude_files: [".gz$"]


level: debug

review: 1

Multiline options

#multiline.pattern: '^[[0-9]{4}-[0-9]{2}-[0-9]{2}'

#multiline.negate: true

#multiline.match: after

#================================ General =====================================

tags: ["test"]

env: staging

#================================ Outputs =====================================

#-------------------------- Elasticsearch output ------------------------------

Array of hosts to connect to.

hosts: [""]

Optional protocol and basic auth credentials.

protocol: "http"
username: "shadow"
password: "Goodluckanrui!"

#----------------------------- Logstash output --------------------------------

The Logstash hosts

#hosts: ["localhost:5044"]

Optional SSL. By default is off.

List of root certificates for HTTPS server verifications

#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

Certificate for SSL client authentication

#ssl.certificate: "/etc/pki/client/cert.pem"

Client Certificate Key

#ssl.key: "/etc/pki/client/cert.key"

#================================ Logging =====================================

Sets log level. The default log level is info.

Available log levels are: critical, error, warning, info, debug

logging.level: debug

At debug level, you can selectively enable logging only for some components.

To enable all selectors use ["*"]. Examples of other selectors are "beat",

"publish", "service".

#logging.selectors: ["*"]

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.