Metricset 'postgresql/bgwriter' is not registered, module not found


(Alberto Gonzalez) #1

I have configure metricbeat to send postgresql metric but get this error when starting filebeat:

2016/09/20 13:54:29.102566 metricbeat.go:26: INFO Register [ModuleFactory:[], MetricSetFactory:[apache/status, mongodb/status, mysql/status, nginx/stubstatus, redis/info, redis/keyspace, system/core, system/cpu, system/diskio, system/filesystem, system/fsstat, system/load, system/memory, system/network, system/process, zookeeper/mntr]]
2016/09/20 13:54:29.201255 beat.go:284: CRIT Exiting: 1 error: metricset 'postgresql/bgwriter' is not registered, module not found
Exiting: 1 error: metricset 'postgresql/bgwriter' is not registered, module not found

config

- module: postgresql
  metricsets:
    # Stats about every PostgreSQL database
#    - database

    # Stats about the background writer process's activity
    - bgwriter

    # Stats about every PostgreSQL process
#    - activity

#  enabled: true
  period: 60s

  # The host must be passed as PostgreSQL DSN. Example:
  # postgres://pqgotest:password@localhost:5432?sslmode=disable
  # The available parameters are documented here:
  # https://godoc.org/github.com/lib/pq#hdr-Connection_String_Parameters
  hosts: ["postgres://postgres@localhost:5432"]

(ruflin) #2

Which version of metricbeat are you using?


(Alberto Gonzalez) #3

5.0-alpha5


(Andrew Kroh) #4

The postgresql module wasn't part of Metricbeat 5.0-alpha5. It will be part of the next release.


(ruflin) #5

If you like to test the postgres module you can find here the snapshot builds: https://beats-nightlies.s3.amazonaws.com/index.html?prefix=metricbeat/


(Alberto Gonzalez) #6

I just see there 6.0-alpha1, no 5.0-alpha6. Is 6.0-alpha1 the right one and compatible with ES 5.0-alpha5?


(ruflin) #7

6.0-alpha1 is currently the right one. We had to do some branch renaming in preparation for beta1. You are fine with this snapshot for the moment to test postgres.


(Alberto Gonzalez) #8

I installed the 6.0-alpha1 but there is no import_dashboards script on /usr/share/metricbeat/kibana

I am using the old dsahboard from old 5.0-alpha5 ... is there a dashboard for postgresql?


(Alberto Gonzalez) #9

Also getting this error on documents:

error QueryStats: pq: unexpected error: "setting PGSERVICEFILE not supported"


(ruflin) #10
  • The import_dashboards script should be under metricbeat/scripts/*
  • There are currently no pre-built dashboards for postgresql. We are more then happy to get external contributions here :wink:

Where exactly do you get this error?


(Alberto Gonzalez) #11

I get the error in Kibana while exploring the documents created by the postgres module


(Andrew Kroh) #12

Could you please provide the JSON document as an example. What version of Postgres are you running?


(Alberto Gonzalez) #13

I am running postgresql 9.2.5 on Centos 6. We cant upgrade it. We have same version on thousends of appliances.

JSON document:

{
  "_index": "metricbeat-2016.09.23",
  "_type": "metricsets",
  "_id": "AVdZCXMREu-Rb_NMhp9v",
  "_score": null,
  "_source": {
    "@timestamp": "2016-09-23T19:52:55.122Z",
    "beat": {
      "hostname": "rc02",
      "name": "rc02"
    },
    "error": "QueryStats: pq: unexpected error: \"setting PGSERVICEFILE not supported\"",
    "fields": {
      "asset_tag": "822-101-50240"
    },
    "metricset": {
      "host": "postgres://postgres@localhost:5432",
      "module": "postgresql",
      "name": "database",
      "rtt": 83
    },
    "postgresql": {
      "database": {}
    },
    "type": "metricsets"
  },
  "fields": {
    "@timestamp": [
      1474660375122
    ],
    "system.network.total.bytes": [
      0
    ]
  },
  "highlight": {
    "metricset.module": [
      "@kibana-highlighted-field@postgresql@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1474660375122
  ]
}

(Andrew Kroh) #14

Does your environment have PGSERVICEFILE set?

From: https://godoc.org/github.com/lib/pq

Most environment variables as specified at http://www.postgresql.org/docs/current/static/libpq-envars.html supported by libpq are also supported by pq. If any of the environment variables not supported by pq are set, pq will panic during connection establishment. Environment variables have a lower precedence than explicitly provided connection parameters.


(Alberto Gonzalez) #15

it is set:

[root@rc02 ~]# echo $PGSERVICEFILE
/etc/sysconfig/pgsql/pg_service.conf


(Alberto Gonzalez) #16

any update on this?


(ruflin) #17

@agonzalez As we use the https://godoc.org/github.com/lib/pq as our library, the only workaround seems to be to remove the PGSERVICEFILE environment variable.


(Alberto Gonzalez) #18

I edited /etc/init.d/metricbeat and added:

unset -v PGSERVICEFILE
unset -v PGSYSCONFDIR

Now i am getting other message error:

QueryStats: pq: unsupported sslmode "allow"; only "require" (default), "verify-full", "verify-ca", and "disable" supported

Any idea?


(Andrew Kroh) #19

That's probably caused by having PGSSLMODE=allow in your environment. I'd unset that one as well. It appears the the Golang lib/pq doesn't agree with the C libpq on what the options are for this setting.

Ref: conn.go
libpq docs


(Alberto Gonzalez) #20

thanks! that worked i am getting now the metrics.

Shouldnt you include by default in the metricbeat init.d the commands to unset the vars?

Is there an easy way to extend it to collect other info from postgresql? like runing a QUERY and sending the result to elk?