I am ashamed to admin that I spent a week or so trying to get SSL enabled in my stack and this is my last ditch effort. I have scoured the docs, message boards, git but I am getting nowhere.
ELK Stack version 7.13.2 installed via Helm
TLS for transport but SSL is not working for HTTP transactions.
Certs have been generated via elasticsearch-certutil and mounted as Kubernetes secrets.
Any breadcrumbs will be appreciated.
elasticsearch.yaml
esConfig:
elasticsearch.yml: |
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
cluster.name: elasticsearch
node.name: elasticsearch-master-0
discovery.seed_hosts: ["127.0.0.1"]
network.host: 0.0.0.0
cluster.initial_master_nodes: ["elasticsearch-master-0"]
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: none
xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.transport.ssl.keystore.password: changeme
xpack.security.transport.ssl.truststore.password: changeme
xpack.security.http.ssl.enabled: true
xpack.security.authc.api_key.enabled: true
xpack.security.http.ssl.client_authentication: none
xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.http.ssl.keystore.password: changeme
xpack.security.http.ssl.truststore.password: changeme
extraEnvs:
- name: "ELASTIC_PASSWORD"
value: "changeme"
- name: "ELASTIC_USERNAME"
value: "changeme"
secretMounts:
- name: elastic-certificates
secretName: elastic-certificates #elastic-certificates.p12
path: /usr/share/elasticsearch/config/certs
defaultMode: 0755
- name: elastic-ca
secretName: elastic-ca #elastic-stack-ca.p12
path: /usr/share/elasticsearch/config/ca
defaultMode: 0755
- name: http-certificates
secretName: http-certificates #elastic-certificates.p12
path: /usr/share/elasticsearch/config/certs/http
defaultMode: 0755
- name: elastic-ca-pem
secretName: elastic-ca-pem #elasticsearch-ca.pem
path: /usr/share/kibana/config/ca/pem
kibana.yaml
kibanaConfig:
kibana.yml: |
elasticsearch.username: "kibana_system"
elasticsearch.password: "changeme"
elasticsearch.ssl.certificateAuthorities: ["/usr/share/kibana/config/ca/elasticsearch-ca.pem"]
elasticsearch.hosts: ["https://elasticsearch-master:9200"]
xpack.encryptedSavedObjects.encryptionKey: 'd2Nuvvt6s0ZmxuBxzzTVMs/i73hjTtRZw+q8YnQxytsK'
server.ssl.keystore.path: "/usr/share/kibana/config/certs/kibana/kibana-server.p12"
server.ssl.keystore.password: "changeme"
server.ssl.enabled: true
elasticsearch.ssl.verificationMode: none
server.name: elasticsearch-master-0
extraEnvs:
- name: "ELASTIC_PASSWORD"
value: "changeme"
- name: "ELASTIC_USERNAME"
value: "changeme"
- name: "NODE_OPTIONS"
value: "--max-old-space-size=1800"
secretMounts:
- name: elastic-certificates
secretName: elastic-certificates #elastic-certificates.p12
path: /usr/share/kibana/config/certs
- name: elastic-ca-pem
secretName: elastic-ca-pem #elasticsearch-ca.pem
path: /usr/share/kibana/config/ca
- name: kibana-certificates
secretName: kibana-certificates #kibana-server.p12
path: /usr/share/kibana/config/certs/kibana
Here is the error that I am getting in Kibana. If I remove the SSL bits Kibana and ES communicate just fine.
{"type":"log","@timestamp":"2021-07-22T23:08:35+00:00","tags":["info","plugins-service"],"pid":951,"message":"Plugin \"timelines\" is disabled."}
{"type":"log","@timestamp":"2021-07-22T23:08:35+00:00","tags":["warning","config","deprecation"],"pid":951,"message":"plugins.scanDirs is deprecated and is no longer used"}
{"type":"log","@timestamp":"2021-07-22T23:08:35+00:00","tags":["warning","config","deprecation"],"pid":951,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.\""}
{"type":"log","@timestamp":"2021-07-22T23:08:38+00:00","tags":["info","plugins-system"],"pid":951,"message":"Setting up [106] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,banners,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,share,newsfeed,mapsEms,mapsLegacy,kibanaLegacy,translations,licenseApiGuard,legacyExport,embeddable,uiActionsEnhanced,expressions,charts,esUiShared,bfetch,data,home,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,advancedSettings,savedObjects,visualizations,visTypeTagcloud,visTypeTable,visTypeVislib,visTypeVega,visTypeMetric,visTypeTimelion,features,licenseManagement,watcher,visTypeMarkdown,visTypeXy,tileMap,regionMap,presentationUtil,canvas,graph,timelion,dashboard,dashboardEnhanced,visualize,visTypeTimeseries,inputControlVis,indexPatternManagement,discover,discoverEnhanced,savedObjectsManagement,spaces,security,savedObjectsTagging,lens,reporting,lists,encryptedSavedObjects,dataEnhanced,dashboardMode,cloud,snapshotRestore,upgradeAssistant,fleet,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,enterpriseSearch,beatsManagement,transform,ingestPipelines,fileUpload,maps,fileDataVisualizer,eventLog,actions,alerting,triggersActionsUi,stackAlerts,ruleRegistry,observability,osquery,ml,securitySolution,cases,infra,monitoring,logstash,apm,uptime]"}
{"type":"log","@timestamp":"2021-07-22T23:08:38+00:00","tags":["info","plugins","taskManager"],"pid":951,"message":"TaskManager is identified by the Kibana UUID: efb7963f-2f7c-48f6-bed1-a7a87b6be4ac"}
{"type":"log","@timestamp":"2021-07-22T23:08:39+00:00","tags":["warning","plugins","security","config"],"pid":951,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-07-22T23:08:39+00:00","tags":["warning","plugins","reporting","config"],"pid":951,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command."}
{"type":"log","@timestamp":"2021-07-22T23:08:39+00:00","tags":["warning","plugins","reporting","config"],"pid":951,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux CentOS 8.4.2105\n OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
{"type":"log","@timestamp":"2021-07-22T23:08:39+00:00","tags":["info","plugins","monitoring","monitoring"],"pid":951,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2021-07-22T23:08:40+00:00","tags":["info","savedobjects-service"],"pid":951,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-07-22T23:08:40+00:00","tags":["error","savedobjects-service"],"pid":951,"message":"Unable to retrieve version information from Elasticsearch nodes."}