Metricbeat: TLS Verification Disabled: certificate signed by unknown authority

[Cactus7600 (Van Vangor)]

Hi. I disabled certificate validation within Metricbeat, yet it's still throwing untrusted certificate errors when hitting Elasticsearch.

I'm using these versions:
Elasticsearch: 7.16.2
Metricbeat: 7.16.1

I have the following YAML settings configured:

metricbeat.yml:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
setup.kibana:
  host: "https://server10.domain.com:5601"
  ssl.enabled: true
  ssl.certificate_authorities: ["/etc/metricbeat/internal_ca.pem"]
  ssl.verification_mode: "none"
output.elasticsearch:
  hosts: ["server10.domain.com:9200"]
  protocol: "https"
  #api_key: "id:api_key"
  username: "remote_monitoring_user"
  password: "password"
  ssl.enabled: true
  ssl.certificate_authorities: ["/etc/metricbeat/internal_ca.pem"]
  ssl.verification_mode: "none"
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/metricbeat
  name: metricbeat
  keepfiles: 7
  permissions: 0640

elasticsearch-xpack.yml

- module: elasticsearch
  xpack.enabled: true
  period: 10s
  hosts: ["https://server10.domain.com:9200"]
  username: "remote_monitoring_user"
  password: "password"
  ssl.certificate_authorities: ["/etc/metricbeat/internal_ca.pem"]
  ssl.verification_mode: "none"

I verified that Metricbeat is honoring that configuration setting by checking the log file after restarting the service. Here's the entry that appeared after making the change:

2023-02-05T21:03:28.877Z        WARN    [tls]   tlscommon/tls_config.go:98      SSL/TLS verifications disabled.

So far, so good, right?

However, Metricbeat immediatly throws a lot of untrusted cert errors similar to this one:

2023-02-05T21:03:28.909Z        ERROR   module/wrapper.go:259   Error fetching data for metricset elasticsearch.shard: error determining if connected Elasticsearch node is master: error making http request: Get "https://server10.domain.com:9200/_nodes/_local/nodes": x509: certificate signed by unknown authority

(I obfuscated the server name.)

So what am I overlooking?

Thanks!
Van

P.S. Yes, disabling certificate validation is a bad practice. However, this a dev environment, and I'm just doing some temporary troubleshooting.

[stephenb (Stephen Brown)]

Hi @Cactus7600 welcome to the community!

@Cactus7600

elasticsearch-xpack.yml
hosts: ["https://server10.domain.com:9200"]
ssl.verification_mode: "none"

In the module Are you sure your indentation is correct.

Hard for us to tell since you only show a snippet and did not format your code

That would be the first thing I would check...

[Cactus7600 (Van Vangor)]

Stephen:

Hi. Thanks for the quick response!

You're right. I didn't do justice to the YAML files. I've now updated the original post to include the full (but sanitized) YAML.

Please let me know if you need to see any other config files.

See Ya!
Van

[Cactus7600 (Van Vangor)]

OK, I figured out the issue.

Before I updated any of the YAML files, I made copies of them so that I could quickly roll back if I accidentally hosed things up. These copies were created within the same folder using this naming convention that included a timestamp:

• filename-YYYYMMDD-HHSS.yml

 

For example, here’s what a backup of the metricbeat.yml file looked like:

• /etc/metricbeat/metricbeat-20230204-1415.yml

 

Everything looks sane so far, right? Well, it appears that one or more of the Elasticsearch-related services (such as Metricbeat) were processing those backup YAML files in addition to the standard YAML files. This caused configuration conflicts that resulted in situations such as TLS validation is disabled, yet the Elasticsearch connection was failing because its certificate was signed by an unknown CA.

The solution was to move all of the backup YAML files to a completely different folder. Then the configuration conflict problems suddenly disappeared. Poof! All’s right with the world!

 
 
 

My theory is that the services that exhibited this behavior, such as Metricbeat, may be using some sort of RegEx expression to select the configuration files so that both of these files were being processed:

• metricbeat.yml
• metricbeat-20230204-1415.yml

That’s pure speculation on my part, though.

 
 
 

“Hey Van, why didn’t you just give backup files a different extension such as .bak? That likely would have avoided this whole issue!”

Indeed! Hindsight is great, isn’t it?

Well, I had been an IIS administrator in a previous life, and one of the rules you learn is to never give your config files (e.g. web.config) a different extension. Why? IIS specifically blocks requests for .config files for security reasons.

However, it was common for new admins to backup a web.config file to something like web.config.bak. And now a 3rd party can request that .bak file, read your connection strings and other sensitive information, and then compromise your site.

As a result of years of that discipline, I never rename the extensions of config files for any app.

 
 
 

Anyway, what’s the moral of the story? Move your backup files to a different folder and you will maintain balance in the universe!

I’m just happy that this issue is resolved. Hopefully this will help any other poor sap that happens to stumble on this nuance of how Metricbeat processes config files.

 

See Ya!
Van

[stephenb (Stephen Brown)]

Good and difficult find I ran into a very similar issue with the modules.. took me a long time to find.. and yes my moral is always change the extension too!