If you can telnet, then you likely aren't using the STARTTLS command, so TLS won't even be used. Is TLS available on your SMTP server? Maybe on a different port? Did your SMTP server generate any more specific messages when Kibana tried to connect to send the email? TLS cipher / algorithm / etc requirements? Or is it possible TLS is not configured correctly on your SMTP server?
I believe the way the call to nodemailer is set up is that it may always require a TLS connection inevitably (direct or via STARTTLS), but we do have some special config so that if secure is off and no auth is used, nodemailer will accept a self-signed certificate. There's also a global config option
xpack.actions.rejectUnauthorized which will accept self-signed certificates for all connector https requests.
If your SMTP server only supports plain tcp connections, I'm not sure the email connector will work in Kibana 7.13. We have some code currently scheduled to be available in 7.14 (not yet generally available) that allows SMTP connections to be specified with an
xpack.actions.customHostSettings[n].smtp.ignoreTLS. That should allow a connection over plain tcp sockets.
I suppose another option if your SMTP server can't support TLS is to use an ssh tunnel / terminator in front of your SMTP server, and have the email connector connect to that - in which case you'd need to turn secure on. Not completely positive that would work, but seems like something that could work.