SSL configuration assistance

Hello Team,

We are having a single node ELK configuration, In the same server we have installed Elasticsearch logstash and kibana (8.0.1) (on-premises)

As per the guide if we are installing the Elasticsearch generates own SSL configuration default.

The Elasticsearch certs are,
in /etc/Elasticsearch/certs/
total 24
-rw-rw----. 1 root Elasticsearch 1915 Apr 29 12:33 http_ca.crt
-rw-rw----. 1 root Elasticsearch 10013 Apr 29 12:33 http.p12
-rw-rw----. 1 root Elasticsearch 5822 Apr 29 12:33 transport.p12

i have tried to configure the output.Elasticsearch in metricbeat in the same server,
Metricbeat configuration:

================================== Outputs ===================================

Configure what output to use when sending the data collected by the beat.

---------------------------- Elasticsearch Output ----------------------------

output.Elasticsearch:

Array of hosts to connect to.

hosts: ["https://sgelastic.saint-gobain.com:9200"]

Protocol - either http (default) or https.

#protocol: "https"

Authentication credentials - either API key or username/password.

#api_key: "id:api_key"
username: "metricbeat_sg"
password: "${MB_PWD}"
#ssl.verification_mode: true
ssl.certificate_authorities: ["/etc/Elasticsearch/certs/http_ca.crt"]
ssl.certificate: "/etc/Elasticsearch/certs/transport.p12"
ssl.key: "/etc/Elasticsearch/Elasticsearch.keystore"

elasticesearch.yml configuration:

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------

The following settings, TLS certificates, and keys have been automatically

generated to configure Elasticsearch security features on 29-04-2022 07:03:14

--------------------------------------------------------------------------------

Enable security features

xpack.security.enabled: true

xpack.security.enrollment.enabled: true

Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents

xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12

Enable encryption and mutual authentication between cluster nodes

xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12

Create a new cluster with the current node only

Additional nodes can still join the cluster later

cluster.initial_master_nodes: ["L04ELKChe001"]

Allow HTTP API connections from localhost and local networks

Connections are encrypted and require user authentication

http.host: [local, site]

Allow other nodes to join the cluster from localhost and local networks

Connections are encrypted and mutually authenticated

#transport.host: [local, site]

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

While we try the metricbeat -e the below error i am facing, please guide me to fix this issue
{"log.level":"error","@timestamp":"2022-05-16T15:21:57.508+0530","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":53},"message":"Failed reading certificate file /etc/Elasticsearch/certs/transport.p12: no pem file","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-16T15:21:57.509+0530","log.origin":{"file.name":"instance/beat.go","file.line":458},"message":"metricbeat stopped.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-05-16T15:21:57.509+0530","log.origin":{"file.name":"instance/beat.go","file.line":1023},"message":"Exiting: error initializing publisher: 1 error: no pem file /etc/Elasticsearch/certs/transport.p12 accessing 'output.Elasticsearch' (source:'/etc/metricbeat/metricbeat.yml')","service.name":"metricbeat","ecs.version":"1.6.0"}
Exiting: error initializing publisher: 1 error: no pem file /etc/Elasticsearch/certs/transport.p12 accessing 'output.Elasticsearch' (source:'/etc/metricbeat/metricbeat.yml')
{"log.level":"error","@timestamp":"2022-05-16T15:21:57.508+0530","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":53},"message":"Failed reading certificate file /etc/Elasticsearch/certs/transport.p12: no pem file","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-16T15:21:57.509+0530","log.origin":{"file.name":"instance/beat.go","file.line":458},"message":"metricbeat stopped.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-05-16T15:21:57.509+0530","log.origin":{"file.name":"instance/beat.go","file.line":1023},"message":"Exiting: error initializing publisher: 1 error: no pem file /etc/Elasticsearch/certs/transport.p12 accessing 'output.Elasticsearch' (source:'/etc/metricbeat/metricbeat.yml')","service.name":"metricbeat","ecs.version":"1.6.0"}
Exiting: error initializing publisher: 1 error: no pem file /etc/Elasticsearch/certs/transport.p12 accessing 'output.Elasticsearch' (source:'/etc/metricbeat/metricbeat.yml')

The certs may not being able to be read by the filebeat user

Try
sudo chmod 666 /etc/Elasticsearch/certs/*

sudo chmod 777 /etc/Elasticsearch/certs

No Luck.

{"log.level":"info","@timestamp":"2022-05-16T18:05:08.715+0530","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1093},"message":"Process info","service.name":"metricbeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null},"cwd":"/etc/Elasticsearch","exe":"/usr/share/metricbeat/bin/metricbeat","name":"metricbeat","pid":594367,"ppid":456065,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2022-05-16T18:05:04.810+0530"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-16T18:05:08.715+0530","log.origin":{"file.name":"instance/beat.go","file.line":323},"message":"Setup Beat: metricbeat; Version: 8.1.0","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-16T18:05:08.717+0530","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-05-16T18:05:08.717+0530","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":53},"message":"Failed reading certificate file /etc/Elasticsearch/certs/transport.p12: no pem file","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-16T18:05:08.717+0530","log.origin":{"file.name":"instance/beat.go","file.line":458},"message":"metricbeat stopped.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-05-16T18:05:08.717+0530","log.origin":{"file.name":"instance/beat.go","file.line":1023},"message":"Exiting: error initializing publisher: 1 error: no pem file /etc/Elasticsearch/certs/transport.p12 accessing 'output.Elasticsearch' (source:'/etc/metricbeat/metricbeat.yml')","service.name":"metricbeat","ecs.version":"1.6.0"}
Exiting: error initializing publisher: 1 error: no pem file /etc/Elasticsearch/certs/transport.p12 accessing 'output.Elasticsearch' (source:'/etc/metricbeat/metricbeat.yml')

Your metricbeat configuration is wrong:

ssl.certificate_authorities: ["/etc/Elasticsearch/certs/http_ca.crt"]
ssl.certificate: "/etc/Elasticsearch/certs/transport.p12"
ssl.key: "/etc/Elasticsearch/Elasticsearch.keystore"

and the error tells you why:

ssl.certificate

takes a PEM encoded client certificate as its value, not a PKCS12 keystore.

Do you know that you need TLS client authentication from metricbeat to elasticsearch ? I would assume that you don't and so you can just comment out

#ssl.certificate: "/etc/Elasticsearch/certs/transport.p12"
#ssl.key: "/etc/Elasticsearch/Elasticsearch.keystore"
1 Like

The metricbeat runs on different server and need the TLS authentication between source and destination.

We have signed certificate for kibana, shall i use the same for Elasticsearch and logstash?
Please advise.

Sure, but you don't necessarily need TLS client authentication ( mutual TLS ). Filebeat can authenticate to Elasticsearch in a different set of ways, take a look at the docs in Configure the Elasticsearch output | Filebeat Reference [8.11] | Elastic and Secure communication with Elasticsearch | Filebeat Reference [8.11] | Elastic . The connection will still be encrypted via TLS but filebeat will authenticate itself to Elasticsearch with credentials or an API key, instead of a client cert. This is usually much simpler unless you have specific requirements to use client TLS authentication.

You should probably not re-use client certificates for different clients. If you absolutely want to use client TLS for filebeat ( and not some other form of authentication ) then you first need to set up a PKI realm in Elasticsearch ( PKI user authentication | Elasticsearch Guide [8.11] | Elastic ) and then generate a new client key and cert for filebeat to use.

Hello,

Still we need to communicate via client certificate, as per the document Elasticsearch 8.1.0 have own certificate while we install.
The location of the Elasticsearch certs are ,
/etc/Elasticsearch/certs/
total 24
-rw-rw----. 1 root Elasticsearch 1915 Apr 29 12:33 http_ca.crt
-rw-rw----. 1 root Elasticsearch 10013 Apr 29 12:33 http.p12
-rw-rw----. 1 root Elasticsearch 5822 Apr 29 12:33 transport.p12

If i configure the http.p12 in filebeat from destination server its not communicating , unknown certificate error.

Please advise.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.