Metricbeat/mongodb leaks connections when metricbeat.config.modules.reload.enabled set to true

Elastic Stack Beats
1

Hi,
I just enabled metricbeat's mongodb module (5.6) and suddenly our mongdb run out of file descriptors. I noticed that metricbeat keeps thousands of connections open to the mongodb.

Further investigation showed that if i enable reload of metricbeat modules:
metricbeat.config.modules:
path: /etc/metricbeat/conf.d/*.yml
reload:
enabled: true
period: 10s

then with every probe to mongodb additional 4 connections are created.
If i disable the reload (set to false), there is no connection leak at all and it only keeps 10 connections open.

Anyone experienced this? Is there a way how to prevent the connection leak?
Thanks,
David

2

There was bug fixed in 6.0 related to reloading. Can you enable debug for the cfgfile selector to see if the module is being constantly reloaded. I think we need to backport this fix to the 5.6 branch. This would make any leak that exist get exposed really quickly.

logging.level: debug
logging.selectors: [cfgfile]

Can you please share the config that you are using for the mongodb module?

Additionally I have opened https://github.com/elastic/beats/issues/5688 because I don't see where the session is closed in mongodb metricsets.

3

Here is the mongodb.yml config for the module:

  • module: mongodb
    metricsets:
    - status
    - dbstats
    enabled: true
    period: 10s
    hosts:
    - mongodb://127.0.0.1:27017
    username: xxxxxxxx
    password: xxxxxxxx

I also tried the connection string with ?maxPoolSize=4 (or other numbers and did not help).

The main metricbeat.yml contains:
metricbeat.config.modules:
path: /etc/metricbeat/conf.d/*.yml
reload:
enabled: true
period: 10s

4

And the log after enabling the debugging says (and yes, the connections stay open):
2017-11-23T06:28:09Z INFO Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat]
2017-11-23T06:28:09Z INFO Setup Beat: metricbeat; Version: 5.6.4
2017-11-23T06:28:09Z INFO Metrics logging every 30s
2017-11-23T06:28:09Z INFO Max Retries set to: 3
2017-11-23T06:28:10Z INFO Activated logstash as output plugin.
2017-11-23T06:28:10Z INFO Publisher name: prod-mongo1
2017-11-23T06:28:10Z INFO Flush Interval set to: 1s
2017-11-23T06:28:10Z INFO Max Bulk Size set to: 2048
2017-11-23T06:28:10Z INFO Register [ModuleFactory:[docker, mongodb, mysql, postgresql, system], MetricSetFactory:[apache/status, ceph/cluster_disk, ceph/cluster_health, ceph/monitor_health, ceph/pool_disk, couchbase/bucket, couchbase/cluster, couchbase/node, docker/container, docker/cpu, docker/diskio, docker/healthcheck, docker/image, docker/info, docker/memory, docker/network, haproxy/info, haproxy/stat, jolokia/jmx, kafka/consumergroup, kafka/partition, mongodb/dbstats, mongodb/status, mysql/status, nginx/stubstatus, php_fpm/pool, postgresql/activity, postgresql/bgwriter, postgresql/database, prometheus/collector, prometheus/stats, redis/info, redis/keyspace, system/core, system/cpu, system/diskio, system/filesystem, system/fsstat, system/load, system/memory, system/network, system/process, system/socket, zookeeper/mntr]]
2017-11-23T06:28:10Z INFO metricbeat start running.
2017-11-23T06:28:10Z WARN BETA: feature dynamic configuration reloading is enabled.
2017-11-23T06:28:10Z INFO Config reloader started
2017-11-23T06:28:20Z DBG Scan for new config files
2017-11-23T06:28:20Z DBG Load config from file: /etc/metricbeat/conf.d/all.yml
2017-11-23T06:28:20Z DBG Load config from file: /etc/metricbeat/conf.d/mongodb.yml
2017-11-23T06:28:20Z DBG Number of module configs found: 2
2017-11-23T06:28:20Z DBG Remove module from stoplist: 1187029132801020061
2017-11-23T06:28:20Z DBG Add module to startlist: 1187029132801020061
2017-11-23T06:28:20Z INFO Connecting to MongoDB node at [127.0.0.1:27017]
2017-11-23T06:28:20Z WARN EXPERIMENTAL: The mongodb dbstats metricset is experimental
2017-11-23T06:28:20Z INFO Connecting to MongoDB node at [127.0.0.1:27017]
2017-11-23T06:28:20Z DBG Remove module from stoplist: 765976364690901654
2017-11-23T06:28:20Z DBG Add module to startlist: 765976364690901654
2017-11-23T06:28:20Z INFO Stopping 0 runners ...
2017-11-23T06:28:20Z INFO Starting 2 runners ...
2017-11-23T06:28:20Z DBG New runner started: 1187029132801020061
2017-11-23T06:28:20Z DBG New runner started: 765976364690901654
2017-11-23T06:28:30Z DBG Scan for new config files
2017-11-23T06:28:30Z DBG Load config from file: /etc/metricbeat/conf.d/all.yml
2017-11-23T06:28:30Z DBG Load config from file: /etc/metricbeat/conf.d/mongodb.yml
2017-11-23T06:28:30Z DBG Number of module configs found: 2
2017-11-23T06:28:30Z DBG Remove module from stoplist: 1187029132801020061
2017-11-23T06:28:30Z INFO Connecting to MongoDB node at [127.0.0.1:27017]
2017-11-23T06:28:30Z WARN EXPERIMENTAL: The mongodb dbstats metricset is experimental
2017-11-23T06:28:30Z INFO Connecting to MongoDB node at [127.0.0.1:27017]
2017-11-23T06:28:30Z DBG Remove module from stoplist: 765976364690901654
2017-11-23T06:28:30Z INFO Stopping 0 runners ...
2017-11-23T06:28:30Z INFO Starting 0 runners ...
2017-11-23T06:28:39Z INFO Non-zero metrics in the last 30s: fetches.mongodb-dbstats.events=10 fetches.mongodb-dbstats.success=2 fetches.mongodb-status.events=2 fetches.mongodb-status.success=2 fetches.system-cpu.events=2 fetches.system-cpu.success=2 fetches.system-diskio.events=6 fetches.system-diskio.success=2 fetches.system-filesystem.events=60 fetches.system-filesystem.success=2 fetches.system-load.events=2 fetches.system-load.success=2 fetches.system-memory.events=2 fetches.system-memory.success=2 fetches.system-network.events=4 fetches.system-network.success=2 fetches.system-process.events=260 fetches.system-process.success=2 libbeat.config.module.running=2 libbeat.config.module.starts=2 libbeat.config.reloads=2 libbeat.logstash.call_count.PublishEvents=2 libbeat.logstash.publish.read_bytes=12 libbeat.logstash.publish.write_bytes=44662 libbeat.logstash.published_and_acked_events=348 libbeat.publisher.messages_in_worker_queues=348 libbeat.publisher.published_events=348
2017-11-23T06:28:40Z DBG Scan for new config files
2017-11-23T06:28:40Z DBG Load config from file: /etc/metricbeat/conf.d/all.yml
2017-11-23T06:28:40Z DBG Load config from file: /etc/metricbeat/conf.d/mongodb.yml
2017-11-23T06:28:40Z DBG Number of module configs found: 2
2017-11-23T06:28:40Z DBG Remove module from stoplist: 1187029132801020061
2017-11-23T06:28:40Z INFO Connecting to MongoDB node at [127.0.0.1:27017]
2017-11-23T06:28:40Z WARN EXPERIMENTAL: The mongodb dbstats metricset is experimental
2017-11-23T06:28:40Z INFO Connecting to MongoDB node at [127.0.0.1:27017]
2017-11-23T06:28:40Z DBG Remove module from stoplist: 765976364690901654
2017-11-23T06:28:40Z INFO Stopping 0 runners ...
2017-11-23T06:28:40Z INFO Starting 0 runners ...
2017-11-23T06:28:50Z DBG Scan for new config files
...

5

This PR should potentially fix the leak: https://github.com/elastic/beats/pull/5711

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.