Hi Everyone,
So am currently using a basic version of ES and just activated Xpack to start using security on my requests but am now receiving ElasticSearch exception [type= security_exception, reason missing authentication token for REST request [/user/account/_search]]
Any ideas why that is?
Basic license doesn;t offer security. By "activated Xpack" , could you mean that you started a trial license? Also, have you gone through our Documentation ?
This looks like a request without authentication credentials which throws this Exception as security is now enabled.
How are you executing this request? If it came from a Browser, then you'd be prompted for authentication (Basic Authentication) . If you're using i.e. curl , you need to pass the credentials with your request with -u username:password
Hi, going via java application having set up the rest high level client to use the cacert and then running the request in postman with basic auth username password
For your Java application you need to specify credentials when you build the low-level client instance (that you then pass to the high-level client) as described here: https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/_basic_authentication.html
For postman, you need to pass credentials with your requests, see Basic Auth
Cheers will give that a go, how would you swap that out for Client certificates?
For the High Level rest client see the last comment here that references HttpAsyncClientBuilder where you can pass the keystore parameter.
For postman, I am not aware how this is done, you can probably find the answer in their docs.
still getting the same issue here's my java code
@Bean
public RestHighLevelClient restHighLevelClient() throws CertificateException, UnrecoverableKeyException, NoSuchAlgorithmException, KeyManagementException, KeyStoreException, URISyntaxException, IOException {
RestClientBuilder builder = getRestClientBuilder();
return new RestHighLevelClient(builder);
}
private RestClientBuilder getRestClientBuilder() throws KeyManagementException, NoSuchAlgorithmException, KeyStoreException, UnrecoverableKeyException, IOException, URISyntaxException, CertificateException {
String storePass = "password";
URL trustStorePathURL = this.getClass().getClassLoader().getResource("keystore/truststore.jks");
assert trustStorePathURL != null;
Path trustStorePath = Paths.get(trustStorePathURL.toURI());
KeyStore trustStore = KeyStore.getInstance("jks");
try (InputStream is = Files.newInputStream(trustStorePath)){
trustStore.load(is, storePass.toCharArray());
}
URL keyStorePathURL = this.getClass().getClassLoader().getResource("keystore/keystore.jks");
assert keyStorePathURL != null;
Path keyStorePath = Paths.get(keyStorePathURL.toURI());
KeyStore keyStore = KeyStore.getInstance("jks");
try (InputStream is = Files.newInputStream(keyStorePath)){
keyStore.load(is, storePass.toCharArray());
}
SSLContextBuilder sslContextBuilder = SSLContexts.custom().loadTrustMaterial(trustStore,null)
.loadKeyMaterial(keyStore,null);
final SSLContext sslContext = sslContextBuilder.build();
return RestClient.builder(new HttpHost("localhost",9200,"https"))
.setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setSSLContext(sslContext));
}Please share a large enough portion of Elasticsearch logs, otherwise we can only guess.
Please do go through the documentation that I linked above and see how Authentication works in Elasticsearch, it would be much helpful.
If you want to use client certificates for authentication, you need to enable and configure a PKI Realm in Elasticsearch. If you have configured Elasticsearch, please share your configuration.
Skimmed through it, I don't see anything particularly wrong with it, but I can't verify it should work as this is not reproducible outside your specific environment.
Error is in the java console will get the logs
Yeah its the 30 day trail
here is the config
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes:
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#
# TLS encryption between nodes
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.key: /etc/elasticsearch/xpack/tls/cert.key
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.certificate: /etc/elasticsearch/xpack/tls/cert.crt
xpack.security.transport.ssl.certificate_authorities: ["/etc/elasticsearch/xpack/tls/cert.crt"]
xpack.security.transport.ssl.client_authentication: optional
#
# TLS HTTP client communication
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /etc/elasticsearch/xpack/tls/cert.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/xpack/tls/cert.crt
xpack.security.http.ssl.certificate_authorities: ["/etc/elasticsearch/xpack/tls/cert.crt"]
#
#
# REALM Config for AD and PKI auth
xpack.security.authc.realms.active_dir.type: active_directory
xpack.security.authc.realms.active_dir.order: 0
xpack.security.authc.realms.active_dir.ssl.certificate: /etc/elasticsearch/xpack/tls/cert.crt
xpack.security.authc.realms.active_dir.ssl.certificate_authorities: ["/etc/elasticsearch/xpack/tls/cert.crt"]
xpack.security.authc.realms.active_dir.ssl.key: /etc/elasticsearch/xpack/tls/cert.key
xpack.security.authc.realms.active_dir.ssl.verification_mode: certificate
xpack.security.authc.realms.default.type: file
xpack.security.authc.realms.default.order: 1
xpack.security.authc.realms.pki.type: pki
xpack.security.authc.realms.pki.order: 2
xpack.security.authc.realms.pki.ssl.certificate_authorities: ["/etc/elasticsearch/xpack/tls/cert.crt"]
Not sure what you mean with the above, but I assume you will post some logs from the error ( Both from your app and Elasticsearch would be optimal ) as otherwise there is not much we can do.
Please don't post images of text as they are hard to read, may not display correctly for everyone, and not searchable.
Instead paste the text and format it with </> icon, and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.
It would be great if you could update your post to solve this.
updated that post will get the errors on java and elasticsearch now
Here is the java output
ErrorCode: UNEXCPECTED_EXCEPTION. ErrorMessage: Elasticsearch exception [type=security_exception, reason=missing authentication token for REST request [/users/profile/_search]]ErrorDetails: Elasticsearch exception [type=security_exception, reason=missing authentication token for REST request [/users/profile/_search]]
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=missing authentication token for REST request [/users/profile/_search]]
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java.177)
at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:573)
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:549)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:456)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:429)
at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:368)
at calvin.es.development.ESSearchService.search(ESSearchService.java:76)
at calvin.es.development.ESSearchService.search(ESSearchService.java:30)
at calvin.es.development.SearchRquestHandle.handleRequest(SearchRquestHandle.java:50)
at calvin.es.development.SearchRquestHandle.handleRequest(SearchRquestHandle.java:23)
at calvin.es.development.MessageRecieverSuper.process(MessageRecieverSuper.java:77)<4 internal calls>
Suppressed: org.elasticsearch.client.ResponseExecption: method [GET], host [https://localhost:9200],URI [/users/profile/_search], status line [HTTP/1.1 401 Unauthorized]
{"error": {"root_cause"}: [{"type":"security_exception","reason":"missing authentication token for REST request [/users/profile/_search]", "header": {"WWW-Authenticate": "Basic realm=\"security\" charset=\"UTF-8\""}}], "type": "security_exception", "reason": "missing authentication token for REST request[/users/profile/_search]","header":{"WWW-Authenticate": "Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:357)
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:346)
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.InputReady(AbstractIODispatch.java:121)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.excute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.excute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
... 1 common frames omitted
Elastic output is same as this line in the java output
Suppressed: org.elasticsearch.client.ResponseExecption: method [GET], host [https://localhost:9200],URI [/users/profile/_search], status line [HTTP/1.1 401 Unauthorized]
{"error": {"root_cause"}: [{"type":"security_exception","reason":"missing authentication token for REST request [/users/profile/_search]", "header": {"WWW-Authenticate": "Basic realm=\"security\" charset=\"UTF-8\""}}], "type": "security_exception", "reason": "missing authentication token for REST request[/users/profile/_search]","header":{"WWW-Authenticate": "Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}You need to set xpack.security.http.ssl.client_authentication to either optional or required as instructed in the documentation I have shared earlier:
The transport or http interface must request client certificates by setting
client_authenticationtooptionalorrequired.
You have set
xpack.security.http.ssl.certificate_authorities: ["/etc/elasticsearch/xpack/tls/cert.crt"]
which means that your clients can only use that specific self signed certificate for authentication. You don't specify what's in the JKS keystore that you use in your code, but I'd thought I'd mention it should contain your cert.key and cert.crt (with your current, configuration that is)
When your Elasticsearch configuration is completed, please try to verify it with curl as such
curl --cert cert.crt --key cert.key --cacert cert.crt 'https://localhost:9200/_xpack/security/_authenticate?pretty'
before focusing again on your client side code.
I made them changes to the config
running
curl --cert cert.crt --key cert.key --cacert cert.crt 'https://localhost:9200/_xpack/security/_authenticate?pretty'
returns
{
"username": "calvin",
"roles" : [
"elastic-read"
],
"full_name" : null,
"email": null,
"metadata" : {
"pki_dn" : "CN = local"
},
"enabled" : true }
client side still showing the same error
Sorry, but your response is rather laconic.
sorry edited that now
I can't understand how both the following lines
"username": "calvin",
and
"pki_dn" : "CN = local"
are part of the same response, as the username for the PKI realm is the CN of the DN that it reads from the client certifiate ( CN = local ) in your case.
openssl x509 -in cert.crt -text -noout
logger.authc.name = org.elasticsearch.xpack.security.authc
logger.authc.level = DEBUG
Sorry typed that response out between machines as i was testing something the actual output was
{
"username": "calvin",
"roles" : [
"elastic-read"
],
"full_name" : null,
"email": null,
"metadata" : {
"pki_dn" : "CN = calvin"
},
"enabled" : true}
To create the keystore and truststore for the client side am using
truststore
keytool -importcert -keystore truststore.jks -file cacert.crt -alias cacert
keystore
keytool -importcert -keystore keystore.jks -file calvin.crt -alias clcertYou need to also add your private key to your keystore, the certificate is not enough. The current state would explain why your client can't authenticate.
Unfortunately keytool doesn't offer functionality for importing a key and certificate pair so you need to create a PKCS#12 out of your pair first
openssl pkcs12 -export -incert.crt -inkey cert.key -out cert.p12 -name cert
(When prompted above, add a password to the PKCS#12 store as the next command depends on it.)
and then import that PKCS#12 store into a JKS keystore with
keytool -importkeystore -destkeystore keystore.jks -srckeystore cert.p12 -srcstoretype PKCS12
Now use that keystore.jks as your keystore in your client code.
In case the above doesn't solve your issue, please follow my suggestions from above:
still no luck
Open SSL command
openssl pkcs12 -export -in cert.crt -inkey cert.key -out cert.p12 -name "cert" -password pass:password
Keytool command
keytool -importkeystore -deststorepass password -destkeystore truststore.jks -srckeystore cert.p12
the truststore.jks is then moved into the resource folder in the java application
Java Code
@Bean
public RestHighLevelClient restHighLevelClient() throws CertificateException, UnrecoverableKeyException, NoSuchAlgorithmException, KeyManagementException, KeyStoreException, URISyntaxException, IOException {
RestClientBuilder builder = getRestClientBuilder();
return new RestHighLevelClient(builder);
}
private RestClientBuilder getRestClientBuilder() throws KeyManagementException, NoSuchAlgorithmException, KeyStoreException, UnrecoverableKeyException, IOException, URISyntaxException, CertificateException {
String storePass = "password";
URL trustStorePathURL = this.getClass().getClassLoader().getResource("keystore/truststore.jks");
Path trustStorePath = Paths.get(trustStorePathURL.toURI());
KeyStore trustStore = KeyStore.getInstance("jks");
try (InputStream is = Files.newInputStream(trustStorePath)){
trustStore.load(is, storePass.toCharArray());
}
SSLContextBuilder sslContextBuilder = SSLContexts.custom().loadTrustMaterial(trustStore,null);
final SSLContext sslContext = sslContextBuilder.build();
return RestClient.builder(new HttpHost("localhost",9200,"https"))
.setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setSSLContext(sslContext));
}
Error Java Console
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1478)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:265)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:305)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
... 16 more
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)
... 18 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
at sun.security.validator.Validator.validate(Validator.java:260)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1501)
... 17 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
... 23 common frames omitted© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.