Filebeat not sending ModSecurity log files

[maxxer (Lorenzo Milesi)]

Hi.
I've configured ModSecurity to generate JSON files this way:

SecAuditEngine On
SecAuditLogType Concurrent
SecAuditLogFormat JSON
SecAuditLogStorageDir /data/nginx/log/modsecurity

This means ModSec will generate a lot of files, one for each HTTP request, in the StorageDir folder. Unfortunately FileBeat doesn't seem to play well with these files. I.e. for a file I get the following (after enabling debug):

$ grep 20190717-072843-156334132380.282675 /var/log/filebeat/filebeat.7
2019-07-18T17:04:47.412+0200    DEBUG   [input] log/input.go:404        Check file for harvesting: /data/nginx/log/modsecurity/integration-tsstest/20190717/20190717-0728/20190717-072843-156334132380.282675
2019-07-18T17:04:47.412+0200    DEBUG   [input] log/input.go:494        Update existing file for harvesting: /data/nginx/log/modsecurity/integration-tsstest/20190717/20190717-0728/20190717-072843-156334132380.282675, offset: 2733
2019-07-18T17:04:47.412+0200    DEBUG   [input] log/input.go:548        File didn't change: /data/nginx/log/modsecurity/integration-tsstest/20190717/20190717-0728/20190717-072843-156334132380.282675

I've looked into filebeat options but couldn't find anything possibly related to this problem. I've been suggested to comment out the ignore_older statement in config, I'm still waiting for feedback to understand if it made any improvement.

filebeat.yml related config:

filebeat.inputs:
- type: log
  paths:
  - /data/nginx/log/modsecurity/**
  fields:
    logzio_codec: json
    token: xxx
    type: libModsecurity
    fields_under_root: true
    encoding: utf-8
    #    ignore_older: 3h

thanks

[martinr_ubi (Martin R.)]

does that mean the filebeat logs your sharing with us were obtained while filebeat was configured with ignore_older:3h ?

Looks like it but the way you phrase the fact that you changed the config and are now waiting is not clear. To me it looks like the logs are from a filebeat which DOES have an ignore older set but the config snippet you have in your post shows ignore older is commmented out.

Your file is from july 17 and your filebeat log line is from july 18 so ignore_older would have ignored it, as designed, no?

[maxxer (Lorenzo Milesi)]

Indeed I had mistakenly posted the updated config with disabled ignore_older. And indeed the timing of the log file isn't explicative.

I've just made a request to the webserver and this is an excerpt from filebeat debug:

2019-07-19T17:41:22.687+0200	DEBUG	[input]	log/input.go:404	Check file for harvesting: /nginx/log/modsecurity/integration-tsstest/20190719/20190719-1737/20190719-173744-156355066491.220255
2019-07-19T17:41:22.687+0200	DEBUG	[input]	log/input.go:494	Update existing file for harvesting: /nginx/log/modsecurity/integration-tsstest/20190719/20190719-1737/20190719-173744-156355066491.220255, offset: 1134
2019-07-19T17:41:22.687+0200	DEBUG	[input]	log/input.go:546	Harvester for file is still running: /nginx/log/modsecurity/integration-tsstest/20190719/20190719-1737/20190719-173744-156355066491.220255
2019-07-19T17:41:22.952+0200	DEBUG	[harvester]	log/log.go:102	End of file reached: /nginx/log/modsecurity/integration-tsstest/20190719/20190719-1737/20190719-173744-156355066453.166529; Backoff now.
2019-07-19T17:41:23.733+0200	DEBUG	[harvester]	log/log.go:102	End of file reached: /nginx/log/modsecurity/integration-tsstest/20190719/20190719-1737/20190719-173744-156355066491.220255; Backoff now.
2019-07-19T17:41:23.882+0200	DEBUG	[harvester]	log/log.go:102	End of file reached: /nginx/log/modsecurity/integration-tsstest/20190719/20190719-1737/20190719-173752-156355067240.842711; Backoff now.
2019-07-19T17:41:23.945+0200	DEBUG	[harvester]	log/log.go:102	End of file reached: /nginx/log/modsecurity/integration-tsstest/20190719/20190719-1737/20190719-173752-156355067291.518043; Backoff now.

these are the only entries regarding the logs generated by the request I made. Do they mean the files has been handled by filebeat? thanks

[martinr_ubi (Martin R.)]

Sorry not used to reverse engineer from the logs like that. You should check if the content of that file is in elasticsearch. But yeah filebeat appear to be reading it because its backing off after end of file.

[maxxer (Lorenzo Milesi)]

Ended up begin a filebeat.yml indentation issue! fields_under_root, encoding and ignore_older tags should have been one step back.