Hi all,
I'm new to ELK setup and trying to figure out what is the problem with one particular log file.
When I run filebeat in debug mode it seems that it parses the file fine (see output below) but when I search on Kibana (no filters just to show everything) it doesn't come anything up.
What am I missing?
2020-11-10T10:16:13.786+0200 DEBUG [input] log/input.go:205 Start next scan
2020-11-10T10:16:13.786+0200 DEBUG [input] log/input.go:439 Check file for harvesting: /var/www/mydomain.com/log/error.log
2020-11-10T10:16:13.786+0200 DEBUG [input] log/input.go:530 Update existing file for harvesting: /var/www/mydomain.com/log/error.log, offset: 6538
2020-11-10T10:16:13.786+0200 DEBUG [input] log/input.go:539 Resuming harvesting of file: /var/www/mydomain.com/log/error.log, offset: 6538, new size: 6882
2020-11-10T10:16:13.787+0200 DEBUG [harvester] log/harvester.go:575 Set previous offset for file: /var/www/mydomain.com/log/error.log. Offset: 6538
2020-11-10T10:16:13.787+0200 DEBUG [harvester] log/harvester.go:566 Setting offset for file: /var/www/mydomain.com/log/error.log. Offset: 6538
2020-11-10T10:16:13.787+0200 DEBUG [harvester] log/harvester.go:207 Harvester setup successful. Line terminator: 1
2020-11-10T10:16:13.787+0200 DEBUG [acker] beater/acker.go:59 stateful ack {"count": 1}
2020-11-10T10:16:13.787+0200 DEBUG [publisher] pipeline/client.go:221 Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:0001-01-01 00:00:00 +0000 UTC Meta:null Fields:null Private:{Id:native::403467-2049 PrevId: Finished:false Fileinfo:0xc0005ee5b0 Source:/var/www/mydomain.com/log/error.log Offset:6538 Timestamp:2020-11-10 10:16:13.786882248 +0200 EET m=+11.673046318 TTL:-1ns Type:log Meta:map[] FileStateOS:403467-2049 IdentifierName:native} TimeSeries:false}
2020-11-10T10:16:13.787+0200 DEBUG [registrar] registrar/registrar.go:264 Processing 1 events
2020-11-10T10:16:13.787+0200 DEBUG [registrar] registrar/registrar.go:231 Registrar state updates processed. Count: 1
2020-11-10T10:16:13.787+0200 DEBUG [registrar] registrar/registrar.go:201 Registry file updated. 117 active states.
2020-11-10T10:16:13.787+0200 DEBUG [harvester] log/harvester.go:485 Update state: /var/www/mydomain.com/log/error.log, offset: 6538
2020-11-10T10:16:13.788+0200 DEBUG [input] log/input.go:226 input states cleaned up. Before: 1, After: 1, Pending: 0
2020-11-10T10:16:13.788+0200 INFO log/harvester.go:299 Harvester started for file: /var/www/mydomain.com/log/error.log
2020-11-10T10:16:13.788+0200 INFO [detect_null_bytes] debug/debug.go:95 Starting debug reader with a buffer size of 16384 and max failures of 100
2020-11-10T10:16:13.788+0200 DEBUG [harvester] log/harvester.go:495 Drop line as it does not match any of the include patterns [Tue Nov 10 10:16:05.378985 2020] [proxy_fcgi:error] [pid 27028] [client 130.43.4.53:56649] AH01071: Got error 'PHP message: PHP Fatal error: <br>The encoded file <b>/var/www/clients/client1/web51/web/myclientdomain/wp-content/plugins/modirumeb-for-woocommerce2/wc-modirumeb.php</b> is not permissioned for 192.168.0.1 in Unknown on line 0'
2020-11-10T10:16:13.788+0200 DEBUG [acker] beater/acker.go:59 stateful ack {"count": 1}
2020-11-10T10:16:13.788+0200 DEBUG [publisher] pipeline/client.go:221 Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:0001-01-01 00:00:00 +0000 UTC Meta:null Fields:null Private:{Id:native::403467-2049 PrevId: Finished:false Fileinfo:0xc0005ee5b0 Source:/var/www/mydomain.com/log/error.log Offset:6882 Timestamp:2020-11-10 10:16:13.786882248 +0200 EET m=+11.673046318 TTL:-1ns Type:log Meta:map[] FileStateOS:403467-2049 IdentifierName:native} TimeSeries:false}
2020-11-10T10:16:13.788+0200 DEBUG [harvester] log/log.go:107 End of file reached: /var/www/mydomain.com/log/error.log; Backoff now.
2020-11-10T10:16:13.792+0200 DEBUG [registrar] registrar/registrar.go:264 Processing 1 events
2020-11-10T10:16:13.792+0200 DEBUG [registrar] registrar/registrar.go:231 Registrar state updates processed. Count: 1
2020-11-10T10:16:13.792+0200 DEBUG [registrar] registrar/registrar.go:201 Registry file updated. 117 active states.
2020-11-10T10:16:14.789+0200 DEBUG [harvester] log/log.go:107 End of file reached: /var/www/mydomain.com/log/error.log; Backoff now.