Does slack action require that we add certs to keystore?

No, this is not what's happening. Watcher will try to make a request to the url you have set in

xpack.notification.slack:
  account:
    <acount_name>:
      url:

Please share relevant parts of your elasticsearch.yml file so that we can help you further.