I am also in need of an answer on this scenario @pebrc. My setup is exactly as described by @Bingu_Shim, although the issue came about for me when upgrading from 1.0.1 => 1.1.0. With 1.0.1 I am able to supply my own cert for both elastic and kibana and have them both configured with DNS. When trying with 1.1.0, I get the same errors above.
Did you ever find a solution to this?
Can you share your config @lsnyder? The one in the OP should still work, though note that the http.tls.selfSignedCertificate
and .certificate
settings should not be used at the same time. The provided certificate should still take priority, but we will likely add validation to prevent this configuration in the future (as setting a custom certificate disables generation of self signed certs).
Secondly, Kibana should work with custom Elasticsearch certificates, but APM will either need the host names to match or for TLS to be disabled completely. Kibana supports validating everything in the certificate but the host name, while APM does not yet. If you can share the specifics of your issue we can try and troubleshoot it anew.
No I was not able to find the solution.
My setup does not include APM, it is just Elastic + Kibana
Here is my config for elastic and kibana:
# elastic.yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: elasticsearch-prd-uks
spec:
auth: {}
http:
service:
metadata:
creationTimestamp: null
spec:
type: LoadBalancer
tls:
certificate:
secretName: <my-secret-name>
nodeSets:
- config:
node.attr.attr_name: attr_value
node.data: true
node.ingest: true
node.master: true
node.ml: true
xpack.monitoring.collection.enabled: true
xpack.security.authc.realms:
native:
native1:
order: 1
count: 3
name: default
podTemplate:
metadata:
creationTimestamp: null
labels:
foo: bar
spec:
containers:
- env:
- name: ES_JAVA_OPTS
value: -Xms6g -Xmx6g
name: elasticsearch
resources:
limits:
cpu: 7350m
memory: 11Gi
requests:
cpu: 7350m
memory: 11Gi
initContainers:
- command:
- sh
- -c
- sysctl -w vm.max_map_count=262144
name: sysctl
resources: {}
securityContext:
privileged: true
- command:
- sh
- -c
- |
bin/elasticsearch-plugin install --batch repository-azure
name: install-plugins
resources: {}
volumeClaimTemplates:
- metadata:
creationTimestamp: null
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
status: {}
secureSettings:
- secretName: <redacted>
- secretName: <redacted>
transport:
service:
metadata:
creationTimestamp: null
spec: {}
updateStrategy:
changeBudget: {}
version: 7.6.2
# kibana.yaml
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: kibana-prd-uks
namespace: default
spec:
count: 1
elasticsearchRef:
name: elasticsearch-prd-uks
http:
service:
metadata: {}
spec:
type: LoadBalancer
tls:
certificate:
secretName: <my-secret-name>
podTemplate:
metadata: {}
spec: {}
version: 7.6.2
Kibana Logs:
{"type":"log","@timestamp":"2020-05-07T17:05:46Z","tags":["error","elasticsearch","data"],"pid":6,"message":"Request error, retrying\nHEAD https://elasticsearch-prd-uks-es-http.default.svc:9200/.apm-agent-configuration => self signed certificate in certificate chain"}
{"type":"log","@timestamp":"2020-05-07T17:05:46Z","tags":["error","elasticsearch","admin"],"pid":6,"message":"Request error, retrying\nGET https://elasticsearch-prd-uks-es-http.default.svc:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => self signed certificate in certificate chain"}
{"type":"log","@timestamp":"2020-05-07T17:05:46Z","tags":["error","elasticsearch","data"],"pid":6,"message":"Request error, retrying\nGET https://elasticsearch-prd-uks-es-http.default.svc:9200/_xpack => self signed certificate in certificate chain"}
{"type":"log","@timestamp":"2020-05-07T17:05:46Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-prd-uks-es-http.default.svc:9200/"}
{"type":"log","@timestamp":"2020-05-07T17:05:46Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-05-07T17:05:46Z","tags":["error","savedobjects-service"],"pid":6,"message":"Unable to retrieve version information from Elasticsearch nodes."}
{"type":"log","@timestamp":"2020-05-07T17:16:54Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-05-07T17:16:56Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-prd-uks-es-http.default.svc:9200/"}
{"type":"log","@timestamp":"2020-05-07T17:16:56Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-05-07T17:16:59Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-prd-uks-es-http.default.svc:9200/"}
{"type":"log","@timestamp":"2020-05-07T17:16:59Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
{"type":"log","@timestamp":"2020-05-07T17:17:01Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: https://elasticsearch-prd-uks-es-http.default.svc:9200/"}
I just tried this configuration with new released version
- elastic operator : 1.1.0
- Elasticsearch, Kibana : 7.6.2
And got same error as you got
Here is kibana error
2020-05-09T08:24:14.077382373Z {"type":"log","@timestamp":"2020-05-09T08:24:14Z","tags":["info","plugins-service"],"pid":7,"message":"Plugin \"case\" is disabled."}
2020-05-09T08:25:29.443788733Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins-system"],"pid":7,"message":"Setting up [37] plugins: [infra,taskManager,siem,licensing,encryptedSavedObjects,code,usageCollection,metrics,canvas,timelion,features,security,apm_oss,translations,reporting,uiActions,data,navigation,status_page,share,newsfeed,kibana_legacy,management,dev_tools,inspector,expressions,visualizations,embeddable,advancedUiActions,dashboard_embeddable_container,home,spaces,cloud,apm,graph,eui_utils,bfetch]"}
2020-05-09T08:25:29.444798641Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","infra"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.447690701Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","taskManager"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.7385755Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","siem"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.739647503Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","licensing"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.744117332Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","encryptedSavedObjects"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.745404276Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["warning","plugins","encryptedSavedObjects","config"],"pid":7,"message":"Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml"}
2020-05-09T08:25:29.84346468Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","code"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.844729461Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","usageCollection"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.847087285Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","metrics"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.847812528Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","canvas"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.853667359Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","timelion"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.854613642Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","features"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.855590132Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","security"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.874897033Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","apm_oss"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.875407265Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","translations"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.876092198Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","data"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.938860566Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","share"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.940414124Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","home"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.945554172Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","spaces"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.950075428Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","cloud"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.95119985Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","apm"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.955940613Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","graph"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.958959265Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","plugins","bfetch"],"pid":7,"message":"Setting up plugin"}
2020-05-09T08:25:29.965921138Z {"type":"log","@timestamp":"2020-05-09T08:25:29Z","tags":["info","savedobjects-service"],"pid":7,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
2020-05-09T08:25:30.061930512Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["error","elasticsearch","data"],"pid":7,"message":"Request error, retrying\nHEAD https://quickstart-es-http.default.svc:9200/.apm-agent-configuration => unable to verify the first certificate"}
2020-05-09T08:25:30.135797692Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["error","elasticsearch","admin"],"pid":7,"message":"Request error, retrying\nGET https://quickstart-es-http.default.svc:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => unable to verify the first certificate"}
2020-05-09T08:25:30.142034057Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["error","elasticsearch","data"],"pid":7,"message":"Request error, retrying\nGET https://quickstart-es-http.default.svc:9200/_xpack => unable to verify the first certificate"}
2020-05-09T08:25:30.150702631Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:30.151361924Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:30.153163439Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["error","savedobjects-service"],"pid":7,"message":"Unable to retrieve version information from Elasticsearch nodes."}
2020-05-09T08:25:30.158642587Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["warning","elasticsearch","data"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:30.159443419Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["warning","elasticsearch","data"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:30.160227781Z Could not create APM Agent configuration: No Living connections
2020-05-09T08:25:30.161510651Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["warning","elasticsearch","data"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:30.162179092Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["warning","elasticsearch","data"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:30.162943621Z {"type":"log","@timestamp":"2020-05-09T08:25:30Z","tags":["warning","plugins","licensing"],"pid":7,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
2020-05-09T08:25:32.5542814Z {"type":"log","@timestamp":"2020-05-09T08:25:32Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:32.554928902Z {"type":"log","@timestamp":"2020-05-09T08:25:32Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:35.052432592Z {"type":"log","@timestamp":"2020-05-09T08:25:35Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:35.052826311Z {"type":"log","@timestamp":"2020-05-09T08:25:35Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:37.555127441Z {"type":"log","@timestamp":"2020-05-09T08:25:37Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:37.555801219Z {"type":"log","@timestamp":"2020-05-09T08:25:37Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:40.058631235Z {"type":"log","@timestamp":"2020-05-09T08:25:40Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:40.05908746Z {"type":"log","@timestamp":"2020-05-09T08:25:40Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:42.555943297Z {"type":"log","@timestamp":"2020-05-09T08:25:42Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:42.556386251Z {"type":"log","@timestamp":"2020-05-09T08:25:42Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:45.059285085Z {"type":"log","@timestamp":"2020-05-09T08:25:45Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:45.059762989Z {"type":"log","@timestamp":"2020-05-09T08:25:45Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:47.559567891Z {"type":"log","@timestamp":"2020-05-09T08:25:47Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:47.560105532Z {"type":"log","@timestamp":"2020-05-09T08:25:47Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
2020-05-09T08:25:50.060813521Z {"type":"log","@timestamp":"2020-05-09T08:25:50Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
2020-05-09T08:25:50.061180134Z {"type":"log","@timestamp":"2020-05-09T08:25:50Z","tags":["warning","elasticsearch","admin"],"pid":7,"message":"No living connections"}
And this is yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: quickstart
spec:
http:
service:
spec:
type: LoadBalancer
loadBalancerIP: <my ip>
tls:
certificate:
secretName: <my secret>
version: 7.6.2
nodeSets:
- name: node
count: 2
config:
node.master: true
node.ingest: true
node.data: true
node.store.allow_mmap: true
podTemplate:
metadata:
labels:
name: node
annotations:
"co.elastic.logs/module": elasticsearch
"co.elastic.metrics/module": elasticsearch
"co.elastic.metrics/period": "10s"
"co.elastic.metrics/hosts": "${data.host}:80"
spec:
initContainers:
- name: sysctl
securityContext:
privileged: true
command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
containers:
- name: elasticsearch
resources:
requests:
memory: 4Gi
cpu: 1
limits:
memory: 4Gi
cpu: 1
env:
- name: ES_JAVA_OPTS
value: "-Xms2g -Xmx2g"
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 200Gi
storageClassName: standard
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: quickstart
spec:
version: 7.6.2
count: 1
elasticsearchRef:
name: quickstart
http:
service:
spec:
type: LoadBalancer
loadBalancerIP: <my ip>
tls:
certificate:
secretName: <my secrets>
podTemplate:
spec:
containers:
- name: kibana
resources:
requests:
memory: 2Gi
cpu: 500m
limits:
memory: 2Gi
cpu: 500m
env:
- name: ES_JAVA_OPTS
value: "-Xms1g -Xmx1g"
And when I connect to Elasticsearch with Chrome, there is no warning with the certificate.
Are you including the CA in your custom certificate? It is possible that the CA is in the chrome certificate store already but not in the Elasticsearch image's. You can also try setting logging.verbose: true
in the Kibana config to see if there are additional logs.
Yes, I have included the intermediate certificate.
And here is the verbose level logs, there was no additional information for the error.
I guess the problem is Kibana is connecting to the the Elasticsearch using Kubernetes internal domain which is different with my Custom Certificate.
There would be two solution for this problem, i guess
- Adding an additional Ingress that handle external domain and certificate
- Change Kibana to connect Elasticsearch using external domain, as described HERE
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","plugins","bfetch"],"pid":6,"message":"Initializing plugin"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["info","plugins","bfetch"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: usageCollection"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: xpack,cloud"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","legacy-service"],"pid":6,"message":"setting up legacy service"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","root"],"pid":6,"message":"starting root"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","server"],"pid":6,"message":"starting server"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","savedobjects-service"],"pid":6,"message":"Starting SavedObjects service"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: migrations"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["info","savedobjects-service"],"pid":6,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["error","elasticsearch","admin"],"pid":6,"message":"Request error, retrying\nGET https://quickstart-es-http.default.svc:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => unable to verify the first certificate"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["error","elasticsearch","data"],"pid":6,"message":"Request error, retrying\nHEAD https://quickstart-es-http.default.svc:9200/.apm-agent-configuration => unable to verify the first certificate"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["error","elasticsearch","data"],"pid":6,"message":"Request error, retrying\nGET https://quickstart-es-http.default.svc:9200/_xpack => unable to verify the first certificate"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"Unable to revive connection: https://quickstart-es-http.default.svc:9200/"}
{"type":"log","@timestamp":"2020-05-13T01:14:35Z","tags":["warning","elasticsearch","data"],"pid":6,"message":"No living connections"}
Could not create APM Agent configuration: No Living connections
I guess the problem is Kibana is connecting to the the Elasticsearch using Kubernetes internal domain which is different with my Custom Certificate.
We configure Kibana to skip verifying the host name by default (elasticsearch.ssl.verificationMode: certificate
) to enable this use case. Since you are receiving an error it cannot validate the chain, may be worth double checking that you have the whole CA chain provided and it validates on its own (without any pre-installed certificates).
Hello, Anya_Sabo
You are right, it solved.
I misconfigured the certificate.
Thank you very much~!!
@Bingu_Shim what was the misconfiguration on your part? I'm still dealing with the issue. The verbose logging didn't expose any additional errors and I can see from the cert being served in chrome that the intermediate is there too.
I have to set following following 3 certificates
- RootCA.crt, ChainCA1.crt, ChainCA2.crt
But, forgot to add ChainCA2.crt when I create Secret.
@Anya_Sabo I'm noticing slightly different kibana config options between 1.0 and 1.1, could this be the source of my issue?
1.1
ssl:
certificateAuthorities: /usr/share/kibana/config/elasticsearch-certs/ca.crt
verificationMode: certificate
1.0
ssl:
verificationMode: certificate
1.0 does not contain a path to CA. I'm guessing as you said the issue i'm hitting on 1.1 is that my CA is not included in that certificateAuthorities location. My CA and certs are all bundled into the same pem which was used to create my secret using:
kubectl create secret generic <cert-name> --from-file=tls.crt=<name>.pem --from-file=tls.key=<name>.key
Any thoughts or guidance from here? Why would it work in 1.0 and not in 1.1. The cert i'm using is the exact same cert, created in the exact same method.
@lsnyder if you can try adding a ca.crt
key to your secret (with the entire CA chain) as described here: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-tls-certificates.html#k8s-setting-up-your-own-certificate
it should populate it correctly. There may have been an inadvertent change between 1.0.1 and 1.1.0 to begin setting that, but I'm not sure off the top of my head and would need to look into it further.
Thanks for reporting and pointing us in the right direction @lsnyder. It looks like your issue in particular was a regression in 1.1.0. We tracked it down in https://github.com/elastic/cloud-on-k8s/issues/3082 and a fix should be available in the next release. In the meantime adding the CA explicitly should resolve it. Sorry about that.
Awesome! Thank you for staying on top of this and helping us out. I can confirm that explicitly supplying the CA when creating the secret worked. We were able to get everything working last night by doing so.
@lsnyder @Bingu_Shim : It seems, you were able to deploy Elastic and Kibana using ECK and custom certificate. Can one of you please share working yaml and commands:
- elastic and Kibana yaml
- create secrete command or yaml file
Need it to deploy it to Azure (AKS).
Thanks in advance.
Hello @nikunjbanker
I did posted sample yaml HERE
and this is command that I used.
kubectl create secret generic <my secret> --from-file=ca.crt=<Root CA File>.crt --from-file=tls.crt=<my crt file>.crt --from-file=tls.key=<my key file>.key