Enable Alerts and Actions

[elasticforme (Sachin)]

Is there a blog site or is there some one might have wrote step by step instruction on how to turn on Alerts and Actions?

I have read few document and trying to follow this but got completely lost

https://www.elastic.co/guide/en/kibana/current/configuring-tls.html#configuring-tls-kib-es

I am using CA certificate between elastic cluster and using user/password from kibana
but kibana is still using http

Created certificated with following command

bin/elasticsearch-certutil cert --ca elastic-certificates.p12

then copy over that to all elasticcluster nodes
and each of them using

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12

this is working fine.
now I am trying to turn on Kibana Alerting.

[stephenb (Stephen Brown)]

Assuming Kibana and Elasticsearch up and running.

See here.

Navigate to Stack Management / Alerts and Actions
is that not there?

[elasticforme (Sachin)]

it is there but trying to setup that security is where I am having difficulty

[stephenb (Stephen Brown)]

To enable security for both elasitcsearch and kibana follow this tutorial

That message indicates that the communication between Kibana and Elasticsearch is not TLS plus you need to enable SSL on kibana have you followed these steps?

I wrote a short how to secure a single node cluster a while back perhaps that will help

[elasticforme (Sachin)]

What is this error means
I am able to setup ssl certificate for elasticsearch communiction.

this is from kibana log. I can use https://localhost:5601 but it just gives me message "kibana server is not ready yet"

{"type":"log","@timestamp":"2021-02-01T10:22:17Z","tags":["error","elasticsearch","data"],"pid":16047,"message":"[ConnectionError]: self signed certificate in certificate chain"}

and this message

{"type":"log","@timestamp":"2021-02-01T10:24:52Z","tags":["warning","elasticsearch","monitoring"],"pid":16047,"message":"Unable to revive connection: https://perelkm01:9200/"}

this tells me missing some certificate. but then I can connect to cluster from command line

curl -u elastic:changeme -k --cacert /etc/elasticsearch/config/elasticsearch-ca.pem https://localhost:9200

What did I miss

[stephenb (Stephen Brown)]

Hard to tell from only snippets of the logs

But if you refer to ES by

https://perelkm01:9200/

Then you needed to add that to the certificate when you created it...

Certs are certs you got to get it right

You might also turn off ssl verification mode

elasticsearch.ssl.verificationMode

[elasticforme (Sachin)]

@stephenb

Then you needed to add that to the certificate when you created it...

This I have not done. is this means every host in cluster who is kibana needs to be added. got it.
Let me test it out today.

[elasticforme (Sachin)]

made little more progress. now kibana is not dying but i have following error messages.
and https:// gives me only message called "Kibana server is not ready yet"

{"type":"log","@timestamp":"2021-02-02T15:01:35Z","tags":["warning","config","deprecation"],"pid":8940,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.\""}
{"type":"log","@timestamp":"2021-02-02T15:01:35Z","tags":["warning","plugins","security","config"],"pid":8940,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2021-02-02T15:01:35Z","tags":["warning","plugins","reporting","config"],"pid":8940,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2021-02-02T15:01:35Z","tags":["warning","plugins","reporting","config"],"pid":8940,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux Red Hat Linux 7.5 OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."}
{"type":"log","@timestamp":"2021-02-02T15:01:35Z","tags":["info","plugins","monitoring","monitoring"],"pid":8940,"message":"config sourced from: production cluster"}
{"type":"log","@timestamp":"2021-02-02T15:01:36Z","tags":["info","savedobjects-service"],"pid":8940,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2021-02-02T15:01:36Z","tags":["error","elasticsearch","monitoring"],"pid":8940,"message":"Request error, retrying\nGET https://perelkm01:9200/_xpack => self signed certificate in certificate chain"}
{"type":"log","@timestamp":"2021-02-02T15:01:36Z","tags":["warning","elasticsearch","monitoring"],"pid":8940,"message":"Unable to revive connection: https://perelkm01:9200/"}
{"type":"log","@timestamp":"2021-02-02T15:01:36Z","tags":["warning","elasticsearch","monitoring"],"pid":8940,"message":"No living connections"}
{"type":"log","@timestamp":"2021-02-02T15:01:36Z","tags":["warning","plugins","licensing"],"pid":8940,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2021-02-02T15:01:36Z","tags":["warning","plugins","monitoring","monitoring"],"pid":8940,"message":"X-Pack Monitoring Cluster Alerts will not be available: No Living connections"}
{"type":"log","@timestamp":"2021-02-02T15:01:36Z","tags":["error","elasticsearch","data"],"pid":8940,"message":"[ConnectionError]: self signed certificate in certificate chain"}
{"type":"log","@timestamp":"2021-02-02T15:01:36Z","tags":["error","savedobjects-service"],"pid":8940,"message":"Unable to retrieve version information from Elasticsearch nodes."}
{"type":"log","@timestamp":"2021-02-02T15:01:38Z","tags":["error","elasticsearch","data"],"pid":8940,"message":"[ConnectionError]: self signed certificate in certificate chain"}
{"type":"log","@timestamp":"2021-02-02T15:01:41Z","tags":["error","elasticsearch","data"],"pid":8940,"message":"[ConnectionError]: self signed certificate in certificate chain"}
{"type":"log","@timestamp":"2021-02-02T15:01:43Z","tags":["error","elasticsearch","data"],"pid":8940,"message":"[ConnectionError]: self signed certificate in certificate chain"}
{"type":"log","@timestamp":"2021-02-02T15:01:46Z","tags":["error","elasticsearch","data"],"pid":8940,"message":"[ConnectionError]: self signed certificate in certificate chain"}

this is what my config looks like kibana.yml

server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/config/kibana.pem
server.ssl.key: /etc/kibana/config/kibana.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/config/selfca.pem" ]

This is what my elasticsearch.yml file looks like

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/config/perelkm01.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/config/perelkm01.p12

# SSL setup
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/config/perelkm01.p12
xpack.security.http.ssl.truststore.path: /etc/elasticsearch/config/perelkm01.p12

[stephenb (Stephen Brown)]

@elasticforme

https://perelkm01:9200/

can you just curl this

curl -u "elastic:myawesomepassword" --cacert certs/selfca.pem https://perelkm01:9200/

does that work? Debugging certs can be painful elastic follows all the standards so you just need to get it right.

First I would make sure the cluster is up .. this is a single node only right. If you followed my walkthrough it should work...

[elasticforme (Sachin)]

@stephenb

curl -u "elastic:myawesomepassword" --cacert certs/selfca.pem https://perelkm01:9200/

yes it works
but I had to use -k option as insuc

curl -k -u elastic:changeme --cacert selfca.pem https://perelkm01:9200/
{
  "name" : "perelkm01",
  "cluster_name" : "test",
  "cluster_uuid" : "1WjGacI4ToCbNly7U3RMEw",
  "version" : {
    "number" : "7.10.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "1c34507e66d7db1211f66f3513706fdf548736aa",
    "build_date" : "2020-12-05T01:00:33.671820Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

[stephenb (Stephen Brown)]

Well then something is not right with the because when I build up following my directions i run without the -k

curl -u elastic:password --cacert ./certs/selfca.pem https://localhost:9200

curl -u elastic:password --cacert ./certs/selfca.pem https://stephenb-metrics-test-es1:9200

curl -u elastic:password --cacert ./certs/selfca.pem https://10.168.0.81:9200

All 3 work..

{
  "name" : "stephenb-metrics-test-es1",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "sIFMd-G-Spi80z4Roaauqw",
  "version" : {
    "number" : "7.7.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "81a1e9eda8e6183f5237786246f6dced26a10eaf",
    "build_date" : "2020-05-10T19:02:51.169154Z",
    "build_snapshot" : false,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

I set up my cert with all three names / ips

You can see in my instructions

./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --ip 10.168.0.71,127.0.0.1 --dns hostname,localhost
openssl pkcs12 -in elastic-stack-ca.p12 -out selfca.pem -clcerts -nokeys

where my hostname is stephenb-metrics-test-es1 and yours should be perelkm01

Detailed Instructions here

We will be setting up TLS on the node. Setup the certs, we will creating self signed certs using the elasticsearch-certutil tool. Note you will want to create the cert with the internal address, and loopback and a dns entry for localhost . If you have an addition external IP you would add to the ip list as well. We also added localhost and the hostname for the DNS section. We are also going to extract / create the ca for later use, you will need it. We are going to create them and then put them in a certs directory in the elasticsearch config directory.

Just take the defaults for all the question and skip the passwords for the .p12s, you can try that later if you like.

sudo -i
cd /usr/share/elasticsearch ./bin/elasticsearch-certutil ca ./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --ip 10.168.0.71,127.0.0.1 --dns hostname,localhost openssl pkcs12 -in elastic-stack-ca.p12 -out selfca.pem -clcerts -nokeys
mkdir /etc/elasticsearch/certs
chmod 755 /etc/elasticsearch/certs 
mv *.p12 /etc/elasticsearch/certs/. 
mv *.pem /etc/elasticsearch/certs/. 
chmod 664 /etc/elasticsearch/certs/*

[elasticforme (Sachin)]

yes I have been following your post

this is three node cluster and I believe ca certificate is working as I can connect to elasticcluster from all node

-k is just because it is new curl

curl message if I don't use -k
    If you'd like to turn off curl's verification of the certificate, use
     the -k (or --insecure) option.

but yes I have not put 127.0.0.1 address when I created individual kibana.key and cert

I only use name, full name and IP address.

Let me test it out this tomorrow morning.

[stephenb (Stephen Brown)]

Can we just say it together... self signed certs are painful
Good luck you will find it.

[elasticforme (Sachin)]

Absolutely agree. it is main pain in..... had spend so much time already trying to figure out.

main reason I want to do this is to activate alert. once successful will have to do in bigger cluster. I will defiantly put document like you did here for someone else to understand. LOL

[elasticforme (Sachin)]

Just to update this thread. Last night I have successfully turn on TLS and HTTPS.
This is three node cluster that I was testing. and I have repetedly done this to make sure procedure works.

@stephenb Thank you for your blog post. it is pretty much using that and some other post. I will post full document in day or two.

[elasticforme (Sachin)]

Full document on setting TLS/HTTPS and SSL

this happens in few steps

1. create internal password
2. create instance file
3. create keys/certs
4. update elasticsearch.yml file
5. restart all
6. check if it worked
7. copy require kesy/certs to kibana locations
8. update kibana.yml file
9. start kiabna and test.

/usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive

I am using all short name.

instances:
  - name: "node01"
    ip:
      - "10.82.1.61"
      - "127.0.0.1"
    dns:
      - "node01"
      - "localhost"
  - name: "node02"
    ip:
      - "10.82.1.62"
      - "127.0.0.1"
    dns:
      - "node02"
      - "localhost"
  - name: "node03"
    ip:
      - "10.82.1.63"
      - "127.0.0.1"
    dns:
      - "node03"
      - "localhost"

/usr/share/elasticsearch/bin/elasticsearch-certutil cert --keep-ca-key --pem --in /root/instance.yml --out /etc/elasticsearch/sachin_certs/certs.zip

unzip certs.zip

Copy all node keys/certs to ca dir for simplicity

rsync  -av node01/* ca/ 
rsync  -av node02/* ca/ 
rsync  -av node03/* ca/ 

copy this ca dir to two more node

rsync  -av * node02:/etc/elasticsearch/sachin_certs/
rsync  -av * node03:/etc/elasticsearch/sachin_certs/

Now setup elasticsearch.yml file, do this on all three node, replace node01 to other node name

xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.key: /etc/elasticsearch/sachin_certs/ca/node01.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/sachin_certs/ca/node01.crt
xpack.security.http.ssl.certificate_authorities: /etc/elasticsearch/sachin_certs/ca/ca.crt
xpack.security.transport.ssl.key: /etc/elasticsearch/sachin_certs/ca/node01.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/sachin_certs/ca/node01.crt
xpack.security.transport.ssl.certificate_authorities: /etc/elasticsearch/sachin_certs/ca/ca.crt
xpack.security.transport.ssl.verification_mode: certificate

Start elasticsearch on all node
test if you have connection. check log files etc..
if this didn't work then go back and check everything. mostly probably you will have typo or something.

curl --cacert /etc/elasticsearch/sachin_certs/ca/ca.crt -u elastic:changeme https://node01:9200/_cat/nodes?v

Now to kibana. it is easy
copy whole certs to kibana config dir.

rsync -av * /etc/kibana/sachin_certs/

change kiaban.yml

server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/sachin_certs/ca/node01.crt
server.ssl.key: /etc/kibana/sachin_certs/ca/node01.key
elasticsearch.hosts: ["https://node01:9200"]
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/sachin_certs/ca/ca.crt" ]
xpack.encryptedSavedObjects.encryptionKey: "something_at_least_32_characters"
elasticsearch.username: "kibana_system"
elasticsearch.password: "kibana_system"

Start kibana and you should be able to connect

https://node01:5601

[stephenb (Stephen Brown)]

@elasticforme Very Nice Thanks!