TL;DR: Yes Elasticsearch supports both encrypted and unencrypted PKCS#8 encoded keys (using the PEM file format).
Longer Explanation:
Certificate format terminology is complicated and can get confusing.
I'll try to be as clear as I can below, but please ask if something doesn't make sense. Or you can skip it, because it's more detail than you probably need, but these topics frequently confuse people so I figured it was worth writing a proper answer...
Strictly speaking, you're not asking about "types of certificates" you're asking about file formats for certificates and keys.
The "certificate type" would actually be "X.509", which is largely redundant because 99.99+% of TLS traffic uses X.509 certificates and most people have never encountered any other type of certificate.
Within X.509 people sometimes talk about the cryptographic algorithms as the "certificate type", so you have "RSA" "DSA" and "ECDSA" key types, and "SHA-1", "SHA-2" and "SHA-256" hashes. A few years ago there was a push from browser vendors to stop trusting "SHA-1 certificates", by which they mean "X.509 certificates that use the SHA-1 hash as part of the signature algorithm".
I don't say any of that to be pedantic, but because it's helpful to be clear on the terminology while searching for answers, so you know what to search for and whether information is relevant or not.
We recommend RSA + SHA-256 certificates, but will support whatever algorithms are supported by your JVM.
PEM is a file format for representing a DER encoded cryptographic object in plain ASCII.
That probably doesn't mean very much, but the short of it is that a PEM file can contain a certificate, but it can also contain a private key, and on occasion other object types as well.
And when there are multiple different DER encodings for a particular object type (and this is true for private keys), it is possible to format each of them as PEM file. But, consequently if a piece of software claims to support "PEM", that gives you no guarantees about which object types & encodings it actually supports.
What this means is, if you have a file that starts with -----BEGIN <object type>----- and ends with -----END <object type>-----, then it's a PEM file, and Elasticsearch can probably handle it but it does depend slightly on what that <object type> is, and what algorithms are used within the object.
PKCS#8 is a standard for encoding private key information using DER. (Technically the standard is for an encoding using ASN.1, which can then be serialised using DER, but now we're getting into very low level semantics).
That means PKCS8 isn't technically a certificate type because it
- is about keys not certificates
- is an encoding format, not a "type" (different key types such as RSA, DSA, and ECDSA can all be encoded into PKCS8).
It also means that technically, PKCS#8 isn't even a file format, because the PKCS#8 standard just tells you how to turn a key into an ASN.1 structure, you still need a way to save that ASN.1 into a file format. And the most common format for that is ... PEM.
Elasticsearch can read PEM formated, PKCS#8 encoded private keys (with or without encryption). However, the PKCS#8 format is quite flexible, and so it's possible to create keys that Elasticsearch can read, but cannot actually use (e.g. because they use an encryption algorithm for which we do not have an implementation).
It is highly unlikely that you will have a PKCS#8 key that Elasticsearch cannot use, but it is possible.