Error establishing direct connection to mongo node at [ip:port]. Error output: no reachable servers

Elastic Stack Beats
1

Hi,

I am using mongodb with replicaset and ssl enabled. When I am trying to create metricbeat mongodb module am not sure How to give URL. I have only CA file. I don't undestand how to solve it.

hosts: ["mongodb://rspelk:eQ5Dn4LR4Z@IP:PORT/admin?authSource=admin&replicaSet=mongodb-replica","IP:PORT/admin?authSource=admin&replicaSet=mongodb-replica","IP:PORT/admin?authSource=admin&replicaSet=mongodb-replica" ]

error am getting in debug logs

Error establishing direct connection to mongo node at [IP:PORT]. Error output: no reachable servers

errror in stdout is

"error": {
"message": "error creating new Session: no reachable servers"
},

Am I doing something wrong. Please help me.

2

Hi! Could you paste your config file here please? If you can also put the full debug log here, that would be helpful too :slightly_smiling_face: Thank you!

3

Hi Kaiyan,

Sorry for late reply.

Below is the Mongodb module that am using.

- module: mongodb
  metricsets:
    - dbstats
#    - status
#    - collstats
#    - metrics
#    - replstatus
  period: 10

  hosts: ["https://username:password@IP1:32000,IP2:32001,IP3:32002/?authSource=admin&replicaSet=mongodb-env5"]
# Optional SSL. By default is off.
  ssl.enabled: true

  # Mode of verification of server certificate ('none' or 'full')
  ssl.verification_mode: 'none'

  # List of root certificates for TLS server verifications
#  ssl.certificate_authorities: "/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/ca.pem"
#  ssl.certificate_authorities: "/etc/logstash/conf.d/ca.pem"
  # Certificate for SSL client authentication
  ssl.certificate: "/etc/metricbeat/env5-0.pem"
4

Debug logs

debug logs

2020-06-03T06:11:27.829Z        INFO    instance/beat.go:621    Home path: [/usr/share/metricbeat/bin] Config path: [/usr/share/metricbeat/bin] Data path: [/usr/share/metricbeat/bin/data] Logs path: [/usr/share/metricbeat/bin/logs]
2020-06-03T06:11:27.830Z        DEBUG   [beat]  instance/beat.go:673    Beat metadata path: /usr/share/metricbeat/bin/data/meta.json
2020-06-03T06:11:27.831Z        INFO    instance/beat.go:629    Beat ID: 3f5d3bfe-c9c6-400a-b940-218d1badb967
2020-06-03T06:11:27.831Z        DEBUG   [seccomp]       seccomp/seccomp.go:117  Loading syscall filter  {"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","clock_gettime","clone","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchown","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}}}
2020-06-03T06:11:27.833Z        INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
2020-06-03T06:11:27.833Z        INFO    [beat]  instance/beat.go:957    Beat info       {"system_info": {"beat": {"path": {"config": "/usr/share/metricbeat/bin", "data": "/usr/share/metricbeat/bin/data", "home": "/usr/share/metricbeat/bin", "logs": "/usr/share/metricbeat/bin/logs"}, "type": "metricbeat", "uuid": "3f5d3bfe-c9c6-400a-b940-218d1badb967"}}}
2020-06-03T06:11:27.839Z        INFO    [beat]  instance/beat.go:966    Build info      {"system_info": {"build": {"commit": "5e69e25b920e3d93bec76a09a31da3ab35a55607", "libbeat": "7.7.0", "time": "2020-05-12T00:57:24.000Z", "version": "7.7.0"}}}
2020-06-03T06:11:27.839Z        INFO    [beat]  instance/beat.go:969    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.13.9"}}}
2020-06-03T06:11:27.842Z        INFO    [beat]  instance/beat.go:973    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-04-23T07:33:15Z","containerized":false,"name":"nodemaster-0","ip":["127.0.0.1/8","::1/128","10.155.195.5/24","fe80::20d:3aff:feaa:d036/64","172.17.0.1/16"],"kernel_version":"4.15.0-1077-azure","mac":["00:0d:3a:aa:d0:36","02:42:9d:c5:e6:13","00:0d:3a:aa:d0:36"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"16.04.6 LTS (Xenial Xerus)","major":16,"minor":4,"patch":6,"codename":"xenial"},"timezone":"UTC","timezone_offset_sec":0,"id":"6d14b0e66c4e4871ba41f690a6668cfa"}}}
2020-06-03T06:11:27.842Z        INFO    [beat]  instance/beat.go:1002   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/root", "exe": "/usr/share/metricbeat/bin/metricbeat", "name": "metricbeat", "pid": 94004, "ppid": 58861, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-06-03T06:11:26.550Z"}}}
2020-06-03T06:11:27.842Z        INFO    instance/beat.go:297    Setup Beat: metricbeat; Version: 7.7.0
2020-06-03T06:11:27.842Z        DEBUG   [beat]  instance/beat.go:323    Initializing output plugins
2020-06-03T06:11:27.843Z        DEBUG   [publisher]     pipeline/consumer.go:137        start pipeline event consumer
2020-06-03T06:11:27.843Z        INFO    [publisher]     pipeline/module.go:110  Beat name: nodemaster-0
2020-06-03T06:11:27.844Z        DEBUG   [registry.lightmodules] mb/lightmodules.go:262  Light modules directory '%!d(string=/usr/share/metricbeat/bin/module)' doesn't exist
2020-06-03T06:11:27.844Z        DEBUG   [modules]       beater/metricbeat.go:148        Available modules and metricsets: Register [ModuleFactory:[aws, azure, beat, cloudfoundry, docker, elasticsearch, kibana, logstash, mongodb, mssql, mysql, oracle, postgresql, system, uwsgi], MetricSetFactory:[aerospike/namespace, apache/status, appsearch/stats, aws/cloudwatch, aws/ec2, aws/rds, aws/s3_daily_storage, aws/s3_request, aws/sqs, azure/compute_vm, azure/compute_vm_scaleset, azure/monitor, azure/storage, beat/state, beat/stats, ceph/cluster_disk, ceph/cluster_health, ceph/cluster_status, ceph/mgr_cluster_disk, ceph/mgr_cluster_health, ceph/mgr_osd_perf, ceph/mgr_osd_pool_stats, ceph/mgr_osd_tree, ceph/mgr_pool_disk, ceph/monitor_health, ceph/osd_df, ceph/osd_tree, ceph/pool_disk, cloudfoundry/container, cloudfoundry/counter, cloudfoundry/value, consul/agent, coredns/stats, couchbase/bucket, couchbase/cluster, couchbase/node, couchdb/server, docker/container, docker/cpu, docker/diskio, docker/event, docker/healthcheck, docker/image, docker/info, docker/memory, docker/network, dropwizard/collector, elasticsearch/ccr, elasticsearch/cluster_stats, elasticsearch/enrich, elasticsearch/index, elasticsearch/index_recovery, elasticsearch/index_summary, elasticsearch/ml_job, elasticsearch/node, elasticsearch/node_stats, elasticsearch/pending_tasks, elasticsearch/shard, envoyproxy/server, etcd/leader, etcd/metrics, etcd/self, etcd/store, golang/expvar, golang/heap, googlecloud/stackdriver, graphite/server, haproxy/info, haproxy/stat, http/json, http/server, istio/citadel, istio/galley, istio/mesh, istio/mixer, istio/pilot, jolokia/jmx, kafka/consumergroup, kafka/partition, kibana/stats, kibana/status, kubernetes/apiserver, kubernetes/container, kubernetes/controllermanager, kubernetes/event, kubernetes/node, kubernetes/pod, kubernetes/proxy, kubernetes/scheduler, kubernetes/state_container, kubernetes/state_cronjob, kubernetes/state_deployment, kubernetes/state_node, kubernetes/state_persistentvolume, kubernetes/state_persistentvolumeclaim, kubernetes/state_pod, kubernetes/state_replicaset, kubernetes/state_resourcequota, kubernetes/state_service, kubernetes/state_statefulset, kubernetes/state_storageclass, kubernetes/system, kubernetes/volume, kvm/dommemstat, logstash/node, logstash/node_stats, memcached/stats, mongodb/collstats, mongodb/dbstats, mongodb/metrics, mongodb/replstatus, mongodb/status, mssql/performance, mssql/transaction_log, munin/node, mysql/galera_status, mysql/status, nats/connections, nats/routes, nats/stats, nats/subscriptions, nginx/stubstatus, openmetrics/collector, oracle/performance, oracle/tablespace, php_fpm/pool, php_fpm/process, postgresql/activity, postgresql/bgwriter, postgresql/database, postgresql/statement, prometheus/collector, prometheus/query, prometheus/remote_write, rabbitmq/connection, rabbitmq/exchange, rabbitmq/node, rabbitmq/queue, redis/info, redis/key, redis/keyspace, sql/query, stan/channels, stan/stats, stan/subscriptions, statsd/server, system/core, system/cpu, system/diskio, system/entropy, system/filesystem, system/fsstat, system/load, system/memory, system/network, system/network_summary, system/process, system/process_summary, system/raid, system/service, system/socket, system/socket_summary, system/uptime, system/users, traefik/health, uwsgi/status, vsphere/datastore, vsphere/host, vsphere/virtualmachine, zookeeper/connection, zookeeper/mntr, zookeeper/server], LightModules:[LightModules:[]]]
2020-06-03T06:11:27.846Z        DEBUG   [tls]   tlscommon/tls.go:71     tls%!(EXTRA string=loading certificate: %v and key %v, string=/etc/metricbeat/env5-0.pem, string=/etc/metricbeat/env5-0.key)
2020-06-03T06:11:27.846Z        INFO    instance/beat.go:438    metricbeat start running.
2020-06-03T06:11:27.846Z        INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2020-06-03T06:11:27.846Z        DEBUG   [module]        module/wrapper.go:127   Starting Wrapper[name=mongodb, len(metricSetWrappers)=1]
2020-06-03T06:11:27.847Z        DEBUG   [module]        module/wrapper.go:181   mongodb/dbstats will start after 7.319503884s
2020-06-03T06:11:27.847Z        INFO    cfgfile/reload.go:175   Config reloader started
2020-06-03T06:11:35.166Z        DEBUG   [module]        module/wrapper.go:189   Starting metricSetWrapper[module=mongodb, name=dbstats, host=IP1:32000,IP2:32001,IP3:32002]
2020-06-03T06:11:35.168Z        DEBUG   [mongodb]       mongodb/mongodb.go:94   Connecting to MongoDB node at [IP1:32000,IP2:32001,IP3:32002]
2020-06-03T06:11:35.170Z        WARN    mongodb/metricset.go:65 Failed to obtain hostname from ``: address IP1:32000,IP2:32001,IP3:32002: too many colons in address
2020-06-03T06:11:35.170Z        WARN    mongodb/metricset.go:65 Failed to obtain hostname from ``: address IP1:32000,IP2:32001,IP3:32002: too many colons in address
2020-06-03T06:11:35.170Z        WARN    [tls]   tlscommon/tls_config.go:83      SSL/TLS verifications disabled.
2020-06-03T06:11:35.170Z        WARN    [tls]   tlscommon/tls_config.go:83      SSL/TLS verifications disabled.
2020-06-03T06:11:35.170Z        WARN    mongodb/metricset.go:65 Failed to obtain hostname from ``: address IP1:32000,IP2:32001,IP3:32002: too many colons in address
2020-06-03T06:11:35.170Z        WARN    [tls]   tlscommon/tls_config.go:83      SSL/TLS verifications disabled.
2020-06-03T06:11:35.215Z        ERROR   mongodb/mongodb.go:98   Error establishing direct connection to mongo node at [IP1:32000,IP2:32001,IP3:32002]. Error output: server returned error on SASL authentication step: Authentication failed.
2020-06-03T06:11:35.215Z        INFO    module/wrapper.go:259   Error fetching data for metricset mongodb.dbstats: error creating new Session: server returned error on SASL authentication step: Authentication failed.
2020-06-03T06:11:35.215Z        DEBUG   [processors]    processing/processors.go:187    Publish event: {
  "@timestamp": "2020-06-03T06:11:35.166Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.7.0"
  },
  "metricset": {
    "name": "dbstats",
    "period": 10000
  },
  "service": {
    "address": "IP1:32000,IP2:32001,IP3:32002",
    "type": "mongodb"
  },
  "error": {
    "message": "error creating new Session: server returned error on SASL authentication step: Authentication failed."
  },
  "event": {
    "duration": 48631602,
    "dataset": "mongodb.dbstats",
    "module": "mongodb"
  }
5

Thank you! Seems like hosts in the config is the problem.

hosts: ["https://username:password@IP1:32000,IP2:32001,IP3:32002/?authSource=admin&replicaSet=mongodb-env5"]

When configuring the hosts option, the following format for MongoDB URLs is required:

[mongodb://][user:pass@]host[:port][?options]

So maybe for your case, the config should look something like:

- module: mongodb
  metricsets:
    - dbstats
#    - status
#    - collstats
#    - metrics
#    - replstatus
  period: 10

  hosts: ["mongodb://username:password@IP1:32000", "mongodb://username:password@IP2:32001", "mongodb://username:password@IP3:32002?authSource=admin&replicaSet=mongodb-env5"]
# Optional SSL. By default is off.
  ssl.enabled: true

  # Mode of verification of server certificate ('none' or 'full')
  ssl.verification_mode: 'none'

  # List of root certificates for TLS server verifications
#  ssl.certificate_authorities: "/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/ca.pem"
#  ssl.certificate_authorities: "/etc/logstash/conf.d/ca.pem"
  # Certificate for SSL client authentication
  ssl.certificate: "/etc/metricbeat/env5-0.pem"
6

Thanks for the reply. I tried it but doesn't work.

7

@jsoriano Do you know what's the correct way to config hosts here for mongodb module? TIA!!

8

Hey, in principle URLs defined as described here should work: https://godoc.org/gopkg.in/mgo.v2#Dial

9

Thanks Jaime!

10

Btw, there seems to be some authentication error too:

Try to pass the username and password with the explicit options for that, something like this:

- module: mongodb
  ...
  hosts:
    - "mongodb://IP1:32000"
    - "mongodb://IP2:32001"
    - "mongodb://IP3:32002"
  ...
  username: username
  password: password

If nothing works it may also be that the server is requiring some authentication mechanism that is not supported by metricbeat or its client library.

11

Hi Jsoriano,

I tried the way same as you shown just by replacing with my IP's and username and password i tried passing separately. Nothing worked. Is it something related to certificate Issue.
I have only CA file.

12

@chandu5565 could you share the error messages you are having with these configurations?

Also, what authentication method is your server expecting? I wonder if it is using an authentication method not supported yet by Metricbeat :thinking:

13

Hi Jsoriano,

Below is the configuration i used for mongodb module.

- module: mongodb
  metricsets:
    - dbstats
  period: 10
  hosts:
     - "mongodb://IP1:32000"
     - "mongodb://IP2:32001"
     - "mongodb://IP3:32002"
  ssl.enabled: true

  # Mode of verification of server certificate ('none' or 'full')
  ssl.verification_mode: 'none'

  # List of root certificates for TLS server verifications
#  ssl.certificate_authorities: "/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/ca.pem"
  ssl.certificate_authorities: "/var/lib/metricbeat/ca.pem"
  username: 'usernametoconnect'

  # Password to use when connecting to MongoDB. Empty by default.
  password: 'password'

debug logs


2020-06-09T09:06:31.085Z        INFO    cfgfile/reload.go:175   Config reloader started
2020-06-09T09:06:31.085Z        INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2020-06-09T09:06:31.085Z        DEBUG   [module]        module/wrapper.go:181   mongodb/dbstats will start after 1.806340806s
2020-06-09T09:06:32.892Z        DEBUG   [module]        module/wrapper.go:189   Starting metricSetWrapper[module=mongodb, name=dbstats, host=IP1:32000]
2020-06-09T09:06:32.893Z        DEBUG   [mongodb]       mongodb/mongodb.go:94   Connecting to MongoDB node at [IP2:32000]
2020-06-09T09:06:32.894Z        WARN    [tls]   tlscommon/tls_config.go:83      SSL/TLS verifications disabled.
2020-06-09T09:06:32.920Z        ERROR   mongodb/mongodb.go:98   Error establishing direct connection to mongo node at [IP3:32000]. Error output: server returned error on SASL authentication step: Authentication failed.
2020-06-09T09:06:32.920Z        INFO    module/wrapper.go:259   Error fetching data for metricset mongodb.dbstats: error creating new Session: server returned error on SASL authentication step: Authentication failed.
2020-06-09T09:06:32.921Z        DEBUG   [processors]    processing/processors.go:187      Publish event: {
  "@timestamp": "2020-06-09T09:06:32.892Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.7.0"
  },
  "event": {
    "dataset": "mongodb.dbstats",
    "module": "mongodb",
    "duration": 28777040
  },
  "metricset": {
    "name": "dbstats",
    "period": 10000
  },
  "error": {
    "message": "error creating new Session: server returned error on SASL authentication step: Authentication failed."
  },
  "service": {
    "address": "IP3:32000",
    "type": "mongodb"
  },
  "host": {
    "name": "nodemaster-0"
  },
  "agent": {
    "hostname": "nodemaster-0",
    "id": "3f5d3bfe-c9c6-400a-b940-218d1badb967",
    "version": "7.7.0",
    "type": "metricbeat",
    "ephemeral_id": "54562b7c-e14c-481f-b4ac-427b276cd6df"
  },
  "ecs": {
    "version": "1.5.0"
  }
}
2020-06-09T09:06:32.993Z        DEBUG   [module]        module/wrapper.go:189   Starting metricSetWrapper[module=mongodb, name=dbstats, host=IP1:32002]
2020-06-09T09:06:32.993Z        DEBUG   [mongodb]       mongodb/mongodb.go:94   Connecting to MongoDB node at [IP2:32002]
2020-06-09T09:06:32.993Z        WARN    [tls]   tlscommon/tls_config.go:83      SSL/TLS verifications disabled.
2020-06-09T09:06:33.006Z        ERROR   mongodb/mongodb.go:98   Error establishing direct connection to mongo node at [IP3:32002]. Error output: server returned error on SASL authentication step: Authentication failed.
2020-06-09T09:06:33.006Z        INFO    module/wrapper.go:259   Error fetching data for metricset mongodb.dbstats: error creating new Session: server returned error on SASL authentication step: Authentication failed.
2020-06-09T09:06:33.008Z        DEBUG   [processors]    processing/processors.go:187
14

The CA file which am using is self signed. For logstash JDBC plugin TO work with connection string i ass CA to keystore for the node then logstash is able to connect with the connection string.

Same string I tried it in metricbeat It doesn't work. Can you also check whether it is related to Certificate and also can you help me how can I used self signed certificate of mongodb server node in metricbeat.

15

Could you try to set the authentication mechanism in the URL? It seems that go client uses MONGODB-CR by default, but PLAIN seems to be needed for SASL.

- module: mongodb
  metricsets:
    - dbstats
  period: 10
  hosts:
     - "mongodb://IP1:32000?authMechanism=PLAIN"
     - "mongodb://IP2:32001?authMechanism=PLAIN"
     - "mongodb://IP3:32002?authMechanism=PLAIN"
  ssl.enabled: true

  # Mode of verification of server certificate ('none' or 'full')
  ssl.verification_mode: 'none'

  # List of root certificates for TLS server verifications
#  ssl.certificate_authorities: "/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/ca.pem"
  ssl.certificate_authorities: "/var/lib/metricbeat/ca.pem"
  username: 'usernametoconnect'

  # Password to use when connecting to MongoDB. Empty by default.
  password: 'password'
16

Regarding SSL, I think CAs must be passed as a list, so, try with:

  ssl.certificate_authorities:
    - "/var/lib/metricbeat/ca.pem"
17

I tried using authMechanism=PLAIN error is below. We are using [SCRAM-SHA-1] mechanism for our mongodb.

  "error": {
    "message": "error creating new Session: Unsupported mechanism 'PLAIN' on authentication database 'admin'"
  }

when I tried with authMechanism=SCRAM-SHA-1

"error": {
    "message": "error creating new Session: server returned error on SASL authentication step: Authentication failed."
  },
18

After making this change the error is same as below

  "error": {
    "message": "error creating new Session: server returned error on SASL authentication step: Authentication failed."
  }
}
19

Ok, have you tried to use SCRAM-SHA-1 in the config instead of PLAIN?

Though I think that this should be supported by the client library and maybe the one used in Metricbeat doesn't support it :confused:
If with authMechanism=SCRAM-SHA-1 it doesn't work I will create an issue to update the client library.

If you also removed ssl.verification_mode: 'none' I guess that these are good news :slight_smile:

20

Actually I am checking and it seems that the library currently used in Metricbeat supports SCRAM-SHA-1, but it may still be some interaction between the library and Metricbeat.