I had packetbeat running no problem until I introduced SSL into the mix. I changed my server name in the mix of this. Packetbeat is looking for the certificate for "oldeservername" and I need it to be "newservername". The logs generate this as soon as the service is started. Thanks! The log and packeteat yml are below:
2018-01-18T11:39:45-06:00 INFO Elasticsearch url: https://newservername:9200
2018-01-18T11:39:45-06:00 ERR Error connecting to Elasticsearch at https://newservername:9200: Get https://newservername:9200: x509: certificate is valid for oldservername, not newservername
2018-01-18T11:39:45-06:00 CRIT Exiting: Error importing Kibana dashboards: fail to create the Elasticsearch loader: Error creating Elasticsearch client: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch https://newservername:9200: Get https://newservername:9200: x509: certificate is valid for oldservername, not newservername]
My packetbeat yml:
#============================== Network device ================================
# Select the network interface to sniff the data. On Linux, you can use the
# "any" keyword to sniff on all connected interfaces.
packetbeat.interfaces.device: any
#================================== Flows =====================================
# Set `enabled: false` or comment out all options to disable flows reporting.
packetbeat.flows:
# Set network flow timeout. Flow is killed if no packet is received before being
# timed out.
timeout: 30s
# Configure reporting period. If set to -1, only killed flows will be reported
period: 10s
#========================== Transaction protocols =============================
packetbeat.protocols:
- type: icmp
# Enable ICMPv4 and ICMPv6 monitoring. Default: false
enabled: true
- type: amqp
# Configure the ports where to listen for AMQP traffic. You can disable
# the AMQP protocol by commenting out the list of ports.
ports: [5672]
- type: cassandra
#Cassandra port for traffic monitoring.
ports: [9042]
- type: dns
# Configure the ports where to listen for DNS traffic. You can disable
# the DNS protocol by commenting out the list of ports.
ports: [53]
# include_authorities controls whether or not the dns.authorities field
# (authority resource records) is added to messages.
include_authorities: true
# include_additionals controls whether or not the dns.additionals field
# (additional resource records) is added to messages.
include_additionals: true
- type: http
# Configure the ports where to listen for HTTP traffic. You can disable
# the HTTP protocol by commenting out the list of ports.
ports: [80, 8080, 8000, 5000, 8002]
- type: memcache
# Configure the ports where to listen for memcache traffic. You can disable
# the Memcache protocol by commenting out the list of ports.
ports: [11211]
- type: mysql
# Configure the ports where to listen for MySQL traffic. You can disable
# the MySQL protocol by commenting out the list of ports.
ports: [3306]
- type: pgsql
# Configure the ports where to listen for Pgsql traffic. You can disable
# the Pgsql protocol by commenting out the list of ports.
ports: [5432]
- type: redis
# Configure the ports where to listen for Redis traffic. You can disable
# the Redis protocol by commenting out the list of ports.
ports: [6379]
- type: thrift
# Configure the ports where to listen for Thrift-RPC traffic. You can disable
# the Thrift-RPC protocol by commenting out the list of ports.
ports: [9090]
- type: mongodb
# Configure the ports where to listen for MongoDB traffic. You can disable
# the MongoDB protocol by commenting out the list of ports.
ports: [27017]
- type: nfs
# Configure the ports where to listen for NFS traffic. You can disable
# the NFS protocol by commenting out the list of ports.
ports: [2049]
- type: tls
# Configure the ports where to listen for TLS traffic. You can disable
# the TLS protocol by commenting out the list of ports.
ports: [443]
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 3
#============================== Dashboards =====================================
setup.dashboards.enabled: true
#============================== Kibana =====================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
http://localhost:5601/path
host: "elkstack:5601"
host: "elkstack:5601"
username: "elastic"
password: "elastic"
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["newservername:9200"]
# Optional protocol and basic auth credentials.
protocol: "https"
username: "admin"
password: "***********"
Packetbeat isn't really looking for "oldeservername". When it connects to https://newservername:9200 it validates that the certificate provided by the server is valid for the hostname to which it is connecting (e.g. newservername). So this means that the server is using an invalid certificate.
What does curl -v https://newservername:9200 return?
You can use openssl s_client -connect newservername:9200 -showcerts to return the server's certificate.
curl -v https://newservername:9200 returns:
* Rebuilt URL to: https://newservername:9200/
* Trying 10.114.11.12...
* Connected to newservername (ip address) port 9200 (#0)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* found 692 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
* Closing connection 0
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
UPDATE:
I changed over to using PEM certificates vs keystore/truststore. Now I am getting the error below. Similar error but not the same.
ERR Error connecting to Elasticsearch at https://newservername:9200: Get https://newservername:9200: x509: certificate signed by unknown authority
2018-01-19T10:56:22-06:00 CRIT Exiting: Error importing Kibana dashboards: fail to create the Elasticsearch loader: Error creating Elasticsearch client: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch https://newservername:9200: Get https://newservername:9200: x509: certificate signed by unknown authority]
Results of curl -v https://newservername:9200
* Rebuilt URL to: https://elkstack:9200/
* Trying 10.114.11.12...
* Connected to elkstack (10.114.11.12) port 9200 (#0)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* found 692 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
* Closing connection 0
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
If you are using a self-signed certificate (vs one signed by a trusted CA like Let's Encrypt) then you need to configure the client to trust that certificate. This is where output.elasticsearch.ssl.certificate_authorities option needs to be used in order to establish trust of the server's certificate.
That change got the packetbeat service up and running. However, I am getting this error. I know the first two entries are standard checks but, the last one is one I haven't seen yet.
2018-01-19T12:53:56-06:00 INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=24213776 beat.memstats.memory_alloc=21307480 beat.memstats.memory_total=22833040 libbeat.config.module.running=0 libbeat.pipeline.clients=13 libbeat.pipeline.events.active=0
2018-01-19T12:54:26-06:00 INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=24213776 beat.memstats.memory_alloc=21321528 beat.memstats.memory_total=22847088 libbeat.config.module.running=0 libbeat.pipeline.clients=13 libbeat.pipeline.events.active=0
2018-01-19T12:54:26-06:00 CRIT Exiting: Error importing Kibana dashboards: fail to create the Kibana loader: Error creating Kibana client: fail to get the Kibana version:HTTP GET request to /api/status fails: fail to execute the HTTP GET request: Get https://newservername:5601//api/status: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). Response: .
Thank you for your help. I am just error after error on this.
It's failing on the the connection to Kibana. Is Kibana setup to listen on HTTPS? Can you curl the URL it's trying to connect to? It should return a JSON document.
There is a double-slash in the URL path. Not sure why, but I'm guessing it's coming from your configuration.
I do not think it is setup up the listen on HTTPS. I am going to change that. I have been reading what makes it listen HTTPS vs HTTP.
The double slash (//) - I cannot find where that is coming from. I looked in the packetbeat.yml and the kibana.yml and neither have and extra slash at the end of the :5601.
Hey Andrew. I went back to the drawing board with the certificates and the beats are sending data to logstash and then to elasticsearch. However, even with the new certificate setup, I am still getting the same x509 error when trying to run the setup --dashboards command.
Exiting: Error importing Kibana dashboards: fail to create the Kibana loader: Error creating Kibana client: fail to load the TLS config: 1 error: tls: failed to parse private key
I could very easily be confused on the certificate part. Do I need to add in a password for this to decrypt the cert/key? If so, would that setting be setup.kibana.ssl.key_passphrase: ?
If your client key is encrypted then yes, you do need provide the passphrase for the key. Does your key.pem have a header with BEGIN ENCRYPTED PRIVATE KEY?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.