Hi, I started using PostgresSQL Metricbeat module on Kubernetes.
I installed the latest version of Metricbeat via helm (elastic/metricbeat chart).
It runs 3 metricbeat pods which show no errors and I can see the metrics in the generated kibana
I am running my postgres database as a service using a third party tool called KubeDB.
This is my metricbeat configuration:
output.elasticsearch: hosts: ["elasticsearch-master:9200"] setup.kibana.host: "kibana-kibana:5601" setup.dashboards.enabled: true setup.template.enabled: true metricbeat.autodiscover: providers: - type: kubernetes templates: - condition: equals: kubernetes.service.name: "postgres" - condition: equals: kubernetes.namespace: dev config: - module: postgresql enabled: true metricsets: - activity - bgwriter - database - statement period: 30s hosts: ["postgres://postgres:5432?sslmode=disable"] username: xxxx password: xxxx
After metricbeat installs, when i perform 'SELECT * FROM pg_stat_activity' query, I can see that there are about a 60-70 database connections from metricbeat in idle state. The number of connections also varies, sometimes it goes down but sometimes it has a propensity to increase until the database starts throwing 'FATAL: sorry, too many clients already' error. I tried to increase max_connections to 700 in postgres.conf and after some time metricbeat creates enough connections to again cause the error.
This is the shortened list of connections (they repeat like this):
I was just wondering is this normal behavior or I am doing something wrong?
If it is normal, can someone please explain the rationale behind the design. I would assume that it's better to reuse or at least timely close database connections.
Thank you in advance