SSL configuration assistance

Elastic Stack Beats
, ,
1

Hello Team,

We are having a single node ELK configuration, In the same server we have installed Elasticsearch logstash and kibana (8.0.1) (on-premises)

As per the guide if we are installing the Elasticsearch generates own SSL configuration default.

The Elasticsearch certs are,
in /etc/Elasticsearch/certs/
total 24
-rw-rw----. 1 root Elasticsearch 1915 Apr 29 12:33 http_ca.crt
-rw-rw----. 1 root Elasticsearch 10013 Apr 29 12:33 http.p12
-rw-rw----. 1 root Elasticsearch 5822 Apr 29 12:33 transport.p12

i have tried to configure the output.Elasticsearch in metricbeat in the same server,
Metricbeat configuration:

================================== Outputs ===================================

Configure what output to use when sending the data collected by the beat.

---------------------------- Elasticsearch Output ----------------------------

output.Elasticsearch:

Array of hosts to connect to.

hosts: ["https://sgelastic.saint-gobain.com:9200"]

Protocol - either http (default) or https.

#protocol: "https"

Authentication credentials - either API key or username/password.

#api_key: "id:api_key"
username: "metricbeat_sg"
password: "${MB_PWD}"
#ssl.verification_mode: true
ssl.certificate_authorities: ["/etc/Elasticsearch/certs/http_ca.crt"]
ssl.certificate: "/etc/Elasticsearch/certs/transport.p12"
ssl.key: "/etc/Elasticsearch/Elasticsearch.keystore"

elasticesearch.yml configuration:

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------

The following settings, TLS certificates, and keys have been automatically

generated to configure Elasticsearch security features on 29-04-2022 07:03:14

--------------------------------------------------------------------------------

Enable security features

xpack.security.enabled: true

xpack.security.enrollment.enabled: true

Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents

xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12

Enable encryption and mutual authentication between cluster nodes

xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12

Create a new cluster with the current node only

Additional nodes can still join the cluster later

cluster.initial_master_nodes: ["L04ELKChe001"]

Allow HTTP API connections from localhost and local networks

Connections are encrypted and require user authentication

http.host: [local, site]

Allow other nodes to join the cluster from localhost and local networks

Connections are encrypted and mutually authenticated

#transport.host: [local, site]

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

While we try the metricbeat -e the below error i am facing, please guide me to fix this issue
{"log.level":"error","@timestamp":"2022-05-16T15:21:57.508+0530","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":53},"message":"Failed reading certificate file /etc/Elasticsearch/certs/transport.p12: no pem file","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-16T15:21:57.509+0530","log.origin":{"file.name":"instance/beat.go","file.line":458},"message":"metricbeat stopped.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-05-16T15:21:57.509+0530","log.origin":{"file.name":"instance/beat.go","file.line":1023},"message":"Exiting: error initializing publisher: 1 error: no pem file /etc/Elasticsearch/certs/transport.p12 accessing 'output.Elasticsearch' (source:'/etc/metricbeat/metricbeat.yml')","service.name":"metricbeat","ecs.version":"1.6.0"}
Exiting: error initializing publisher: 1 error: no pem file /etc/Elasticsearch/certs/transport.p12 accessing 'output.Elasticsearch' (source:'/etc/metricbeat/metricbeat.yml')
{"log.level":"error","@timestamp":"2022-05-16T15:21:57.508+0530","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":53},"message":"Failed reading certificate file /etc/Elasticsearch/certs/transport.p12: no pem file","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-16T15:21:57.509+0530","log.origin":{"file.name":"instance/beat.go","file.line":458},"message":"metricbeat stopped.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-05-16T15:21:57.509+0530","log.origin":{"file.name":"instance/beat.go","file.line":1023},"message":"Exiting: error initializing publisher: 1 error: no pem file /etc/Elasticsearch/certs/transport.p12 accessing 'output.Elasticsearch' (source:'/etc/metricbeat/metricbeat.yml')","service.name":"metricbeat","ecs.version":"1.6.0"}
Exiting: error initializing publisher: 1 error: no pem file /etc/Elasticsearch/certs/transport.p12 accessing 'output.Elasticsearch' (source:'/etc/metricbeat/metricbeat.yml')

2

The certs may not being able to be read by the filebeat user

Try
sudo chmod 666 /etc/Elasticsearch/certs/*

sudo chmod 777 /etc/Elasticsearch/certs

3

No Luck.

{"log.level":"info","@timestamp":"2022-05-16T18:05:08.715+0530","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1093},"message":"Process info","service.name":"metricbeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null},"cwd":"/etc/Elasticsearch","exe":"/usr/share/metricbeat/bin/metricbeat","name":"metricbeat","pid":594367,"ppid":456065,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2022-05-16T18:05:04.810+0530"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-05-16T18:05:08.715+0530","log.origin":{"file.name":"instance/beat.go","file.line":323},"message":"Setup Beat: metricbeat; Version: 8.1.0","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-05-16T18:05:08.717+0530","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-05-16T18:05:08.717+0530","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":53},"message":"Failed reading certificate file /etc/Elasticsearch/certs/transport.p12: no pem file","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-05-16T18:05:08.717+0530","log.origin":{"file.name":"instance/beat.go","file.line":458},"message":"metricbeat stopped.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-05-16T18:05:08.717+0530","log.origin":{"file.name":"instance/beat.go","file.line":1023},"message":"Exiting: error initializing publisher: 1 error: no pem file /etc/Elasticsearch/certs/transport.p12 accessing 'output.Elasticsearch' (source:'/etc/metricbeat/metricbeat.yml')","service.name":"metricbeat","ecs.version":"1.6.0"}
Exiting: error initializing publisher: 1 error: no pem file /etc/Elasticsearch/certs/transport.p12 accessing 'output.Elasticsearch' (source:'/etc/metricbeat/metricbeat.yml')

4

Your metricbeat configuration is wrong:

ssl.certificate_authorities: ["/etc/Elasticsearch/certs/http_ca.crt"]
ssl.certificate: "/etc/Elasticsearch/certs/transport.p12"
ssl.key: "/etc/Elasticsearch/Elasticsearch.keystore"

and the error tells you why:

ssl.certificate

takes a PEM encoded client certificate as its value, not a PKCS12 keystore.

Do you know that you need TLS client authentication from metricbeat to elasticsearch ? I would assume that you don't and so you can just comment out

#ssl.certificate: "/etc/Elasticsearch/certs/transport.p12"
#ssl.key: "/etc/Elasticsearch/Elasticsearch.keystore"
1 Like
5

The metricbeat runs on different server and need the TLS authentication between source and destination.

We have signed certificate for kibana, shall i use the same for Elasticsearch and logstash?
Please advise.

6

Sure, but you don't necessarily need TLS client authentication ( mutual TLS ). Filebeat can authenticate to Elasticsearch in a different set of ways, take a look at the docs in Configure the Elasticsearch output | Filebeat Reference [8.11] | Elastic and Secure communication with Elasticsearch | Filebeat Reference [8.11] | Elastic . The connection will still be encrypted via TLS but filebeat will authenticate itself to Elasticsearch with credentials or an API key, instead of a client cert. This is usually much simpler unless you have specific requirements to use client TLS authentication.

You should probably not re-use client certificates for different clients. If you absolutely want to use client TLS for filebeat ( and not some other form of authentication ) then you first need to set up a PKI realm in Elasticsearch ( PKI user authentication | Elasticsearch Guide [8.11] | Elastic ) and then generate a new client key and cert for filebeat to use.

7

Hello,

Still we need to communicate via client certificate, as per the document Elasticsearch 8.1.0 have own certificate while we install.
The location of the Elasticsearch certs are ,
/etc/Elasticsearch/certs/
total 24
-rw-rw----. 1 root Elasticsearch 1915 Apr 29 12:33 http_ca.crt
-rw-rw----. 1 root Elasticsearch 10013 Apr 29 12:33 http.p12
-rw-rw----. 1 root Elasticsearch 5822 Apr 29 12:33 transport.p12

If i configure the http.p12 in filebeat from destination server its not communicating , unknown certificate error.

Please advise.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.