hi, there
I was trying to build a filebeat module for naxsi waf log.
the log is on /usr/local/shadow/logs/shadow_waf.log
the log format is below:
2017/09/07 18:26:21 [error] 1097#0: SHADOW_WAF: 109720&ip=172.16.144.1&server=trip.cmbchina.com&uri=/SSO/Common/Login/%26quot%3B../Nest/img/blueline.gif%26quot%3B&learning=0&vers=0.55.3&total_processed=24&total_blocked=6&block=1&cscore0=$SQL&score0=8&cscore1=$XSS&score1=16&zone0=URL&id0=1008&var_name0=&content=''
the grok pattern is from logstash config file (which works alright on logstash):
the field.yml file on waf/_meta direcotry:
- name: waf
type: group
description: >
Contains fields for shadow waf logs.
fields:- name: severity
type: keyword
description: >
severity of log. - name: pid
type: keyword
description: >
process id. - name: tag
type: keyword
description: >
The tag of waf log. - name: support_id
type: keyword
description: >
support id of the attack. - name: client_ip
type: keyword
description: >
client ip address. - name: http_host
type: keyword
description: >
app domain. - name: http_uri
type: keyword
description: >
access url. - name: learning
type: keyword
description: >
the status code of learning. - name: version
type: keyword
description: >
version of shadow waf module. - name: processed
type: keyword
description: >
number of processed requests. - name: total_blocked
type: keyword
description: >
number of total blocked requests. - name: blocked
type: keyword
description: >
number of blocked requests. - name: type
type: keyword
description: >
type of the web attack. - name: score
type: keyword
description: >
the score of the attack. - name: zone
type: keyword
description: >
attack part of the http request. - name: rule_id
type: keyword
description: >
which id of the rule. - name: http_variable
type: keyword
description: >
http variable of the attack. - name: attack_content
type: keyword
description: >
actual attack request and body. - name: geoip
type: group
description: >
Contains GeoIP information gathered based on the client_ip field.
Only present if the GeoIP Elasticsearch plugin is available and
used.
fields:- name: continent_name
type: keyword
description: >
The name of the continent. - name: country_iso_code
type: keyword
description: >
Country ISO code. - name: location
type: geo_point
description: >
The longitude and latitude.
- name: continent_name
- name: severity
the waf.yml on waf/config directory:
type: log
paths:
{{ range $i, $path := .paths }}
- {{path}} {{ end }} exclude_files: [".gz"]
the pipeline.json on shadow/ingest directory:
{
"description": "Pipeline for parsing shadow waf logs.Requires the geoip plugin.",
"processors": [{
"grok": {
"field": "message",
"patterns":[
"(?<shadow.waf.time>%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME}) [%{LOGLEVEL:shadow.waf.severity}] %{POSINT:shadow.waf.pid}#%{NUMBER}: %{WORD:shadow.waf.tag}: %{NUMBER:shadow.waf.support_id}&ip=%{IP:shadow.waf.client_ip}&server=%{IPORHOST:shadow.waf.http_host}&uri=%{GREEDYDATA:shadow.waf.http_uri}&learning=%{WORD:shadow.waf.learning}&vers=%{DATA:shadow.waf.version}&total_processed=%{NUMBER:shadow.waf.processed}&total_blocked=%{NUMBER:shadow.waf.total_blocked}&block=%{NUMBER:shadow.waf.blocked}(&cscore0=%{GREEDYDATA:shadow.waf.type}&score0=%{NUMBER:shadow.waf.score})?(&cscore1=%{GREEDYDATA:shadow.waf.type}&score1=%{NUMBER:shadow.waf.score})?(&cscore2=%{GREEDYDATA:shadow.waf.type}&score2=%{NUMBER:shadow.waf.score})?(&cscore3=%{GREEDYDATA:shadow.waf.type}&score3=%{NUMBER:shadow.waf.score})?(&zone0=%{WORD:shadow.waf.zone}&id0=%{NUMBER:shadow.waf.rule_id}&var_name0=%{USERNAME:shadow.waf.http_variable})?(&zone1=%{WORD:shadow.waf.zone}&id1=%{NUMBER:shadow.waf.rule_id}&var_name1=%{USERNAME:shadow.waf.http_variable}?)?(&zone2=%{WORD:shadow.waf.zone}&id2=%{NUMBER:shadow.waf.rule_id}&var_name2=%{USERNAME:shadow.waf.http_variable}?)?(&zone3=%{WORD:shadow.waf.zone}&id3=%{NUMBER:shadow.waf.rule_id}&var_name3=%{USERNAME:shadow.waf.http_variable}?)?(&content='%{GREEDYDATA:shadow.waf.attack_content}')?"
],
"ignore_missing": true
}
},{
"remove":{
"field": "message"
}
}, {
"rename": {
"field": "@timestamp",
"target_field": "read_timestamp"
}
}, {
"date": {
"field": "shadow.waf.time",
"target_field": "@timestamp",
"formats": ["dd/MMM/YYYY:H:m:s Z"]
}
}, {
"remove": {
"field": "shadow.waf.time"
}
}, {
"geoip": {
"field": "shadow.waf.client_ip",
"target_field": "shadow.waf.geoip"
}
}],
"on_failure" : [{
"set" : {
"field" : "error.message",
"value" : "{{ _ingest.on_failure_message }}"
}
}]
}
the manifest.yml on shadow directory
module_version: 1.0
var:
- name: paths
default:- /usr/local/shadow/logs/shadow_waf.log
os.darwin: - /usr/local/shadow/logs/shadow_waf.log
os.windows: - c:/programdata/shadow/logs/shadow_waf.log
- /usr/local/shadow/logs/shadow_waf.log
ingest_pipeline: ingest/pipeline.json
prospector: config/waf.yml
requires.processors:
- name: geoip
plugin: ingest-geoip
just follow the developer guide, but when I try to use the module on filebeat.yml, the filebeat failed to start with no logs on /var/log/filebeat/filebeat.
any inputs are appreciated!
