Trying to use filebeat postgresql conf.d to push logs to kibana

Andrew_Freitas · July 16, 2020, 9:15am

So thats where, I believe, the issue lies.
Filebeat doesn't seem to be constantly pushing the logs out.
It initially gives something like

Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: {
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "@timestamp": "2020-07-16T09:02:56.714Z",
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "@metadata": {
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "beat": "filebeat",
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "type": "_doc",
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "version": "7.5.1"
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: },
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "agent": {
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "hostname": "DEVCCM13DCTDB01.cpp.nonlive",
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "id": "b90d9dd6-2a7d-459c-86fa-546f7bb236f3",
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "version": "7.5.1",
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "type": "filebeat",
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "ephemeral_id": "e369ff1d-a095-408e-abb0-c45e65d9b5f9"
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: },
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "ecs": {
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "version": "1.1.0"
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: },
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "log": {
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "offset": 39684,
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "file": {
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "path": "/var/log/pg_log/postgresql-Thu.log"
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: }
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: },
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "message": "2020-07-16 09:02:49.264 UTC  @ [30097-475] : LOG:  checkpoint starting: time",
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "fields": {
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "custom_log_type": "postgresql",
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "environment": "dev"
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: },
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "input": {
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "type": "log"
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: },
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "host": {
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "name": "DEVCCM13DCTDB01.cpp.nonlive"
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: }
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: }
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: {
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "@timestamp": "2020-07-16T09:02:56.714Z",
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "@metadata": {
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "beat": "filebeat",
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "type": "_doc",
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "version": "7.5.1"
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: },
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "log": {
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "offset": 39761,
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "file": {
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "path": "/var/log/pg_log/postgresql-Thu.log"
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: }
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: },
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "message": "2020-07-16 09:02:49.886 UTC  @ [30097-476] : LOG:  checkpoint complete: wrote 6 buffers (0.0%); 0 transaction log file(s) added, 0 removed, 0 recycled; write=0.535 s, sync=0.021 s, total=0.622 s; sync files=4, longest=0.009 s, average=0.005 s; distance=16364 kB, estimate=16364 kB",
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "input": {
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "type": "log"
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: },
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "fields": {
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "environment": "dev",
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "custom_log_type": "postgresql"
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: },
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "ecs": {
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "version": "1.1.0"
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: },
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "host": {
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "name": "DEVCCM13DCTDB01.cpp.nonlive"
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: },
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "agent": {
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "type": "filebeat",
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "ephemeral_id": "e369ff1d-a095-408e-abb0-c45e65d9b5f9",
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "hostname": "DEVCCM13DCTDB01.cpp.nonlive",
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "id": "b90d9dd6-2a7d-459c-86fa-546f7bb236f3",
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: "version": "7.5.1"
Jul 16 09:02:57 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: }

Followed by this afterwards, nonstop

[root@DEVCCM13DCTDB01 filebeat]# journalctl -xef --unit filebeat.service | grep postgresql-Thu
Jul 16 09:10:54 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: 2020-07-16T09:10:54.050Z        DEBUG        [input]        log/input.go:511        Update existing file for harvesting: /var/log/pg_log/postgresql-Thu.log, offset: 40397
Jul 16 09:10:54 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: 2020-07-16T09:10:54.050Z        DEBUG        [input]        log/input.go:563        Harvester for file is still running: /var/log/pg_log/postgresql-Thu.log
Jul 16 09:10:56 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: 2020-07-16T09:10:56.737Z        DEBUG        [harvester]        log/log.go:107        End of file reached: /var/log/pg_log/postgresql-Thu.log; Backoff now.
Jul 16 09:10:59 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: 2020-07-16T09:10:59.050Z        DEBUG        [input]        log/input.go:421        Check file for harvesting: /var/log/pg_log/postgresql-Thu.log
Jul 16 09:10:59 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: 2020-07-16T09:10:59.050Z        DEBUG        [input]        log/input.go:511        Update existing file for harvesting: /var/log/pg_log/postgresql-Thu.log, offset: 40397
Jul 16 09:10:59 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: 2020-07-16T09:10:59.050Z        DEBUG        [input]        log/input.go:563        Harvester for file is still running: /var/log/pg_log/postgresql-Thu.log
Jul 16 09:11:04 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: 2020-07-16T09:11:04.051Z        DEBUG        [input]        log/input.go:421        Check file for harvesting: /var/log/pg_log/postgresql-Thu.log
Jul 16 09:11:04 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: 2020-07-16T09:11:04.051Z        DEBUG        [input]        log/input.go:511        Update existing file for harvesting: /var/log/pg_log/postgresql-Thu.log, offset: 40397
Jul 16 09:11:04 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: 2020-07-16T09:11:04.051Z        DEBUG        [input]        log/input.go:563        Harvester for file is still running: /var/log/pg_log/postgresql-Thu.log
Jul 16 09:11:06 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: 2020-07-16T09:11:06.738Z        DEBUG        [harvester]        log/log.go:107        End of file reached: /var/log/pg_log/postgresql-Thu.log; Backoff now.
Jul 16 09:11:09 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: 2020-07-16T09:11:09.052Z        DEBUG        [input]        log/input.go:421        Check file for harvesting: /var/log/pg_log/postgresql-Thu.log
Jul 16 09:11:09 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: 2020-07-16T09:11:09.052Z        DEBUG        [input]        log/input.go:511        Update existing file for harvesting: /var/log/pg_log/postgresql-Thu.log, offset: 40397
Jul 16 09:11:09 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: 2020-07-16T09:11:09.052Z        DEBUG        [input]        log/input.go:563        Harvester for file is still running: /var/log/pg_log/postgresql-Thu.log
Jul 16 09:11:14 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: 2020-07-16T09:11:14.053Z        DEBUG        [input]        log/input.go:421        Check file for harvesting: /var/log/pg_log/postgresql-Thu.log
Jul 16 09:11:14 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: 2020-07-16T09:11:14.053Z        DEBUG        [input]        log/input.go:511        Update existing file for harvesting: /var/log/pg_log/postgresql-Thu.log, offset: 40397
Jul 16 09:11:14 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: 2020-07-16T09:11:14.053Z        DEBUG        [input]        log/input.go:563        Harvester for file is still running: /var/log/pg_log/postgresql-Thu.log
Jul 16 09:11:16 DEVCCM13DCTDB01.cpp.nonlive filebeat[42561]: 2020-07-16T09:11:16.738Z        DEBUG        [harvester]        log/log.go:107        End of file reached: /var/log/pg_log/postgresql-Thu.log; Bac

I suspect extra configuration is required of the filebeat, but not sure what exactly.
Postgres logs about every 5mins.

Topic		Replies	Views
Kibana is not showing all the logfiles from the path, shows only one file Beats filebeat	23	7286	July 5, 2017
6.5.4 Filebeat w enabled postgresql module doesn't create proper index - no postgresql.. fields Beats filebeat	5	1784	April 15, 2019
Kibana output port not accepted in filebeat.yml Beats filebeat	15	1277	December 9, 2019
No matching indices found no indices match pattern filebeat-* Beats filebeat	18	10099	August 14, 2018
The filebeat dashboard : No results found Beats filebeat	14	7428	July 24, 2017

Trying to use filebeat postgresql conf.d to push logs to kibana

Related topics