# Filebeat not sending ModSecurity log files **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [July 18, 2019, 3:14pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-modsecurity-log-files/191224 "2019-07-18T15:14:10Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![maxxer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maxxer/32/50472_2.png) [@maxxer](https://discuss.elastic.co/u/maxxer)\ **Post date:** [July 18, 2019, 3:14pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-modsecurity-log-files/191224/1 "2019-07-18T15:14:10Z")
Hi. I've configured ModSecurity to generate JSON files this way: ```auto SecAuditEngine On SecAuditLogType Concurrent SecAuditLogFormat JSON SecAuditLogStorageDir /data/nginx/log/modsecurity ``` This means ModSec will generate _a lot_ of files, **one for each HTTP request** , in the _StorageDir_ folder. Unfortunately FileBeat doesn't seem to play well with these files. I.e. for a file I get the following (after enabling debug): ```bash $ grep 20190717-072843-156334132380.282675 /var/log/filebeat/filebeat.7 2019-07-18T17:04:47.412+0200 DEBUG [input] log/input.go:404 Check file for harvesting: /data/nginx/log/modsecurity/integration-tsstest/20190717/20190717-0728/20190717-072843-156334132380.282675 2019-07-18T17:04:47.412+0200 DEBUG [input] log/input.go:494 Update existing file for harvesting: /data/nginx/log/modsecurity/integration-tsstest/20190717/20190717-0728/20190717-072843-156334132380.282675, offset: 2733 2019-07-18T17:04:47.412+0200 DEBUG [input] log/input.go:548 File didn't change: /data/nginx/log/modsecurity/integration-tsstest/20190717/20190717-0728/20190717-072843-156334132380.282675 ``` I've looked into [filebeat options](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html) but couldn't find anything possibly related to this problem. I've been suggested to comment out the _ignore\_older_ statement in config, I'm still waiting for feedback to understand if it made any improvement. filebeat.yml related config: ```auto filebeat.inputs: - type: log paths: - /data/nginx/log/modsecurity/** fields: logzio_codec: json token: xxx type: libModsecurity fields_under_root: true encoding: utf-8 # ignore_older: 3h ``` thanks ---
**Author:** ![martinr\_ubi](https://avatars.discourse-cdn.com/v4/letter/m/b5e925/32.png) [@martinr\_ubi](https://discuss.elastic.co/u/martinr_ubi)\ **Post date:** [July 19, 2019, 2:11am UTC](https://discuss.elastic.co/t/filebeat-not-sending-modsecurity-log-files/191224/2 "2019-07-19T02:11:08Z")
does that mean the filebeat logs your sharing with us were obtained while filebeat was configured with ignore\_older:3h ? Looks like it but the way you phrase the fact that you changed the config and are now waiting is not clear. To me it looks like the logs are from a filebeat which DOES have an ignore older set but the config snippet you have in your post shows ignore older is commmented out. Your file is from july 17 and your filebeat log line is from july 18 so ignore\_older would have ignored it, as designed, no? ---
**Author:** ![maxxer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maxxer/32/50472_2.png) [@maxxer](https://discuss.elastic.co/u/maxxer)\ **Post date:** [July 19, 2019, 3:53pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-modsecurity-log-files/191224/3 "2019-07-19T15:53:03Z")
Indeed I had mistakenly posted the _updated_ config with disabled `ignore_older`. And indeed the timing of the log file isn't explicative. I've just made a request to the webserver and this is an excerpt from filebeat debug: ```auto 2019-07-19T17:41:22.687+0200 DEBUG [input] log/input.go:404 Check file for harvesting: /nginx/log/modsecurity/integration-tsstest/20190719/20190719-1737/20190719-173744-156355066491.220255 2019-07-19T17:41:22.687+0200 DEBUG [input] log/input.go:494 Update existing file for harvesting: /nginx/log/modsecurity/integration-tsstest/20190719/20190719-1737/20190719-173744-156355066491.220255, offset: 1134 2019-07-19T17:41:22.687+0200 DEBUG [input] log/input.go:546 Harvester for file is still running: /nginx/log/modsecurity/integration-tsstest/20190719/20190719-1737/20190719-173744-156355066491.220255 2019-07-19T17:41:22.952+0200 DEBUG [harvester] log/log.go:102 End of file reached: /nginx/log/modsecurity/integration-tsstest/20190719/20190719-1737/20190719-173744-156355066453.166529; Backoff now. 2019-07-19T17:41:23.733+0200 DEBUG [harvester] log/log.go:102 End of file reached: /nginx/log/modsecurity/integration-tsstest/20190719/20190719-1737/20190719-173744-156355066491.220255; Backoff now. 2019-07-19T17:41:23.882+0200 DEBUG [harvester] log/log.go:102 End of file reached: /nginx/log/modsecurity/integration-tsstest/20190719/20190719-1737/20190719-173752-156355067240.842711; Backoff now. 2019-07-19T17:41:23.945+0200 DEBUG [harvester] log/log.go:102 End of file reached: /nginx/log/modsecurity/integration-tsstest/20190719/20190719-1737/20190719-173752-156355067291.518043; Backoff now. ``` these are the only entries regarding the logs generated by the request I made. Do they mean the files has been handled by filebeat? thanks ---
**Author:** ![martinr\_ubi](https://avatars.discourse-cdn.com/v4/letter/m/b5e925/32.png) [@martinr\_ubi](https://discuss.elastic.co/u/martinr_ubi)\ **Post date:** [July 19, 2019, 5:11pm UTC](https://discuss.elastic.co/t/filebeat-not-sending-modsecurity-log-files/191224/4 "2019-07-19T17:11:57Z")
Sorry not used to reverse engineer from the logs like that. You should check if the content of that file is in elasticsearch. But yeah filebeat appear to be reading it because its backing off after end of file. ---
**Author:** ![maxxer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maxxer/32/50472_2.png) [@maxxer](https://discuss.elastic.co/u/maxxer)\ **Post date:** [July 23, 2019, 8:53am UTC](https://discuss.elastic.co/t/filebeat-not-sending-modsecurity-log-files/191224/5 "2019-07-23T08:53:56Z")
Ended up begin a filebeat.yml indentation issue! `fields_under_root`, `encoding` and `ignore_older` tags should have been one step back. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 20, 2019, 8:53am UTC](https://discuss.elastic.co/t/filebeat-not-sending-modsecurity-log-files/191224/6 "2019-08-20T08:53:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.