# Filebeat for naxsi waf log **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [November 9, 2017, 1:44am UTC](https://discuss.elastic.co/t/filebeat-for-naxsi-waf-log/106951 "2017-11-09T01:44:00Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![qingweiqm](https://avatars.discourse-cdn.com/v4/letter/q/f4b2a3/32.png) [@qingweiqm](https://discuss.elastic.co/u/qingweiqm)\ **Post date:** [November 9, 2017, 1:44am UTC](https://discuss.elastic.co/t/filebeat-for-naxsi-waf-log/106951/1 "2017-11-09T01:44:00Z")
hi, there I was trying to build a filebeat module for naxsi waf log. the log is on /usr/local/shadow/logs/shadow\_waf.log ## the log format is below: ## 2017/09/07 18:26:21 [error] 1097#0: SHADOW\_WAF: 109720&ip=172.16.144.1&[server=trip.cmbchina.com](http://server=trip.cmbchina.com)&uri=/SSO/Common/Login/%26quot%3B../Nest/img/blueline.gif%26quot%3B&learning=0&vers=0.55.3&total\_processed=24&total\_blocked=6&block=1&cscore0=$SQL&score0=8&cscore1=$XSS&score1=16&zone0=URL&id0=1008&var\_name0=&content='' the grok pattern is from logstash config file (which works alright on logstash): ![46](https://us1.discourse-cdn.com/elastic/original/3X/8/3/8381a8880064e2421c9d5eb7e479f24472ac674c.png) ## the field.yml file on waf/\_meta direcotry: - name: waf type: group description: \> Contains fields for shadow waf logs. fields: - name: severity type: keyword description: \> severity of log. - name: pid type: keyword description: \> process id. - name: tag type: keyword description: \> The tag of waf log. - name: support\_id type: keyword description: \> support id of the attack. - name: client\_ip type: keyword description: \> client ip address. - name: http\_host type: keyword description: \> app domain. - name: http\_uri type: keyword description: \> access url. - name: learning type: keyword description: \> the status code of learning. - name: version type: keyword description: \> version of shadow waf module. - name: processed type: keyword description: \> number of processed requests. - name: total\_blocked type: keyword description: \> number of total blocked requests. - name: blocked type: keyword description: \> number of blocked requests. - name: type type: keyword description: \> type of the web attack. - name: score type: keyword description: \> the score of the attack. - name: zone type: keyword description: \> attack part of the http request. - name: rule\_id type: keyword description: \> which id of the rule. - name: http\_variable type: keyword description: \> http variable of the attack. - name: attack\_content type: keyword description: \> actual attack request and body. - name: geoip type: group description: \> Contains GeoIP information gathered based on the client\_ip field. Only present if the GeoIP Elasticsearch plugin is available and used. fields: - name: continent\_name type: keyword description: \> The name of the continent. - name: country\_iso\_code type: keyword description: \> Country ISO code. - name: location type: geo\_point description: \> The longitude and latitude. * * * ## the waf.yml on waf/config directory: type: log paths: {{ range $i, $path := .paths }} - {{path}} {{ end }} exclude\_files: [".gz"] * * * ## the pipeline.json on shadow/ingest directory: ## { "description": "Pipeline for parsing shadow waf logs.Requires the geoip plugin.", "processors": [{ "grok": { "field": "message", "patterns":["(?\%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[-]%{TIME}) [%{LOGLEVEL:shadow.waf.severity}] %{POSINT:shadow.waf.pid}#%{NUMBER}: %{WORD:shadow.waf.tag}: %{NUMBER:shadow.waf.support\_id}&ip=%{IP:shadow.waf.client\_ip}&server=%{IPORHOST:shadow.waf.http\_host}&uri=%{GREEDYDATA:shadow.waf.http\_uri}&learning=%{WORD:shadow.waf.learning}&vers=%{DATA:shadow.waf.version}&total\_processed=%{NUMBER:shadow.waf.processed}&total\_blocked=%{NUMBER:shadow.waf.total\_blocked}&block=%{NUMBER:shadow.waf.blocked}(&cscore0=%{GREEDYDATA:shadow.waf.type}&score0=%{NUMBER:shadow.waf.score})?(&cscore1=%{GREEDYDATA:shadow.waf.type}&score1=%{NUMBER:shadow.waf.score})?(&cscore2=%{GREEDYDATA:shadow.waf.type}&score2=%{NUMBER:shadow.waf.score})?(&cscore3=%{GREEDYDATA:shadow.waf.type}&score3=%{NUMBER:shadow.waf.score})?(&zone0=%{WORD:shadow.waf.zone}&id0=%{NUMBER:shadow.waf.rule\_id}&var\_name0=%{USERNAME:shadow.waf.http\_variable})?(&zone1=%{WORD:shadow.waf.zone}&id1=%{NUMBER:shadow.waf.rule\_id}&var\_name1=%{USERNAME:shadow.waf.http\_variable}?)?(&zone2=%{WORD:shadow.waf.zone}&id2=%{NUMBER:shadow.waf.rule\_id}&var\_name2=%{USERNAME:shadow.waf.http\_variable}?)?(&zone3=%{WORD:shadow.waf.zone}&id3=%{NUMBER:shadow.waf.rule\_id}&var\_name3=%{USERNAME:shadow.waf.http\_variable}?)?(&content='%{GREEDYDATA:shadow.waf.attack\_content}')?" ], "ignore\_missing": true } },{ "remove":{ "field": "message" } }, { "rename": { "field": "@timestamp", "target\_field": "read\_timestamp" } }, { "date": { "field": "shadow.waf.time", "target\_field": "@timestamp", "formats": ["dd/MMM/YYYY:H:m:s Z"] } }, { "remove": { "field": "shadow.waf.time" } }, { "geoip": { "field": "shadow.waf.client\_ip", "target\_field": "shadow.waf.geoip" } }], "on\_failure" : [{ "set" : { "field" : "error.message", "value" : "{{ \_ingest.on\_failure\_message }}" } }] } ## the manifest.yml on shadow directory module\_version: 1.0 var: - name: paths default: - /usr/local/shadow/logs/shadow\_waf.log os.darwin: - /usr/local/shadow/logs/shadow\_waf.log os.windows: - c:/programdata/shadow/logs/shadow\_waf.log ingest\_pipeline: ingest/pipeline.json prospector: config/waf.yml requires.processors: - name: geoip plugin: ingest-geoip * * * just follow the developer guide, but when I try to use the module on filebeat.yml, the filebeat failed to start with no logs on /var/log/filebeat/filebeat. any inputs are appreciated! ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 9, 2017, 2:36am UTC](https://discuss.elastic.co/t/filebeat-for-naxsi-waf-log/106951/2 "2017-11-09T02:36:35Z")
Please format your code using the `` button, or markdown style back ticks, it's really hard to read as it is now 🙂 ---
**Author:** ![qingweiqm](https://avatars.discourse-cdn.com/v4/letter/q/f4b2a3/32.png) [@qingweiqm](https://discuss.elastic.co/u/qingweiqm)\ **Post date:** [November 9, 2017, 3:04am UTC](https://discuss.elastic.co/t/filebeat-for-naxsi-waf-log/106951/3 "2017-11-09T03:04:19Z")
got it ... thanks ---
**Author:** ![qingweiqm](https://avatars.discourse-cdn.com/v4/letter/q/f4b2a3/32.png) [@qingweiqm](https://discuss.elastic.co/u/qingweiqm)\ **Post date:** [November 10, 2017, 2:11am UTC](https://discuss.elastic.co/t/filebeat-for-naxsi-waf-log/106951/4 "2017-11-10T02:11:35Z")
I just adjust the style, is it OK? ---
**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\ **Post date:** [November 10, 2017, 9:58am UTC](https://discuss.elastic.co/t/filebeat-for-naxsi-waf-log/106951/5 "2017-11-10T09:58:19Z")
Are Grok patterns in the pipeline working? You can test it using Simulate API: [https://www.elastic.co/guide/en/elasticsearch/reference/master/simulate-pipeline-api.html](https://www.elastic.co/guide/en/elasticsearch/reference/master/simulate-pipeline-api.html) Could you please share the log of Filebeat? ---
**Author:** ![qingweiqm](https://avatars.discourse-cdn.com/v4/letter/q/f4b2a3/32.png) [@qingweiqm](https://discuss.elastic.co/u/qingweiqm)\ **Post date:** [November 10, 2017, 2:22pm UTC](https://discuss.elastic.co/t/filebeat-for-naxsi-waf-log/106951/6 "2017-11-10T14:22:11Z")
the grok patterns are working in my logstash conf. when I try to start the filebeat with filebeat.yml which call the new module. It just failed to start with no logs on the /var/log/filebeat/filebeat ## when I replace the filebeat.yml and restart the filebeat, the log is below: ## 2017-11-10T22:20:37+08:00 DBG Received sigterm/sigint, stopping 2017-11-10T22:20:37+08:00 INFO Stopping filebeat 2017-11-10T22:20:37+08:00 INFO Stopping Crawler 2017-11-10T22:20:37+08:00 INFO Stopping 1 prospectors 2017-11-10T22:20:37+08:00 INFO Prospector ticker stopped 2017-11-10T22:20:37+08:00 INFO Stopping Prospector: 5287455123581690130 2017-11-10T22:20:37+08:00 INFO Prospector channel stopped because beat is stopping. 2017-11-10T22:20:37+08:00 INFO Crawler stopped 2017-11-10T22:20:37+08:00 INFO Stopping spooler 2017-11-10T22:20:37+08:00 DBG Spooler has stopped 2017-11-10T22:20:37+08:00 DBG Shutting down sync publisher 2017-11-10T22:20:37+08:00 INFO Stopping Registrar 2017-11-10T22:20:37+08:00 INFO Ending Registrar 2017-11-10T22:20:37+08:00 DBG Write registry file: /var/lib/filebeat/registry 2017-11-10T22:20:37+08:00 DBG Registry file updated. 5 states written. 2017-11-10T22:20:37+08:00 INFO Total non-zero values: libbeat.es.publish.read\_bytes=2662 libbeat.es.publish.read\_errors=1 libbeat.es.publish.write\_bytes=596 publish.events=2 registrar.states.current=5 registrar.states.update=2 registrar.writes=2 2017-11-10T22:20:37+08:00 INFO Uptime: 8m56.879360288s 2017-11-10T22:20:37+08:00 INFO filebeat stopped. ## And here is the filebeat.yml: ###################### Filebeat Configuration Example ######################### filebeat.modules: - module: shadow waf: enbaled: true var.paths: ["/usr/local/shadow/logs/shadow\_waf.log"] # #=========================== Filebeat prospectors ============================= #filebeat.prospectors: #- input\_type: log # paths: # - /usr/local/shadow/logs/shadow.log #fields: #level: shadow #fields\_under\_root: true #exclude\_lines: ["^DBG"] #include\_lines: ["^ERR", "^WARN"] #exclude\_files: [".gz$"] #fields: # level: debug # review: 1 ### Multiline options #multiline.pattern: '^[[0-9]{4}-[0-9]{2}-[0-9]{2}' #multiline.negate: true #multiline.match: after #================================ General ===================================== #name: tags: ["test"] #fields: # env: staging #================================ Outputs ===================================== #-------------------------- Elasticsearch output ------------------------------ output.elasticsearch: # Array of hosts to connect to. hosts: ["172.16.144.167:9201"] # Optional protocol and basic auth credentials. protocol: "http" username: "shadow" password: "Goodluckanrui!" #----------------------------- Logstash output -------------------------------- #output.logstash: # The Logstash hosts #hosts: ["localhost:5044"] # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications #ssl.certificate\_authorities: ["/etc/pki/root/ca.pem"] # Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #ssl.key: "/etc/pki/client/cert.key" #================================ Logging ===================================== # Sets log level. The default log level is info. # Available log levels are: critical, error, warning, info, debug logging.level: debug # At debug level, you can selectively enable logging only for some components. # To enable all selectors use ["\*"]. Examples of other selectors are "beat", # "publish", "service". ## #logging.selectors: ["\*"] ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 8, 2017, 2:22pm UTC](https://discuss.elastic.co/t/filebeat-for-naxsi-waf-log/106951/7 "2017-12-08T14:22:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.