Logstash and Elasticsearch using HTTPS

zpraxis · July 17, 2019, 1:06am

Hi!

I'm having trouble using HTTPS between Logstash and Elasticsearch. I have already enabled xpack.security features via:

xpack.security.enabled: true

And added the corresponding *.transport.* and *.ssl.* properties in elasticsearch.yml.

Also, in Logstash, I have:

output {
    elasticsearch {
      hosts => [ "my_ip_adress:9200" ]
      keystore => "${HOME}/elastic-ops/logstash/config/logstash.p12"
      keystore_password => "${keystore_password}"
      ssl => true
      ssl_certificate_verification => true
      cacert => "${HOME}/elastic-ops/logstash/config/ca.root.crt"
  }
}

For some reason, when I start both of them, I get the following in Logstash log:

[2019-07-16T19:44:03,829][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://my_ip_address:9200/", :error_type=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://my_ip_address:9200/][Manticore::SocketException] Connection reset"}

And in Elasticsearch log I get something like the following:

java.lang.IllegalArgumentException: did not find a SSLContext for [SSLConfiguration{keyConfig=[keyPath=[/home/d_elastic/elastic-ops/elasticsearch/config/node_http.key], certPaths=[/home/d_elastic/elastic-ops/elasticsearch/config/node_http.crt]], trustConfig=ca=[/home/d_elastic/elastic-ops/elasticsearch/config/ca.root.crt]], cipherSuites=[[TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA]], supportedProtocols=[[TLSv1.2, TLSv1.1]], sslClientAuth=[REQUIRED], verificationMode=[FULL]}]
at org.elasticsearch.xpack.core.ssl.SSLService.sslContextHolder(SSLService.java:304) ~[x-pack-core-7.2.0.jar:7.2.0]
at org.elasticsearch.xpack.core.ssl.SSLService.sslContext(SSLService.java:292) ~[x-pack-core-7.2.0.jar:7.2.0]
at org.elasticsearch.xpack.core.ssl.SSLService.createSSLEngine(SSLService.java:248) ~[x-pack-core-7.2.0.jar:7.2.0]
at org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport$HttpSslChannelHandler.initChannel(SecurityNetty4HttpServerTransport.java:81) ~[?:?]
at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:969) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:610) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1461) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1126) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:651) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:515) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:428) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:487) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.35.Final.jar:4.1.35.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:405) [netty-common-4.1.35.Final.jar:4.1.35.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.35.Final.jar:4.1.35.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.35.Final.jar:4.1.35.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

This same stacktrace I get when I try to run for the first time the tool to set the passwords for the built-in users.

What could be happening? Am I missing something?

TimV · July 17, 2019, 5:36am

Can you post your elasticsearch.yml please?

It looks like there's something incorrect in your config, and we're failing in a very unhelpful way.

zpraxis · July 17, 2019, 2:50pm

Thanks for your prompt response!

Aside from basic properties set like paths, the rest of the file is like the following:

cluster.name: my_cluster
node.name: ${HOSTNAME}-node

path.data: /full_path_to_custom_data_dir/data/
path.logs: /full_path_to_custom_logs_dir/logs/

network.host: my_ip
http.port: 9200

discovery.seed_hosts: ["my_ip:9200"]
cluster.initial_master_nodes: ["${HOSTNAME}-node"]

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.key: /full_path_to_custom_config_dir/node.key
xpack.security.transport.ssl.certificate: /full_path_to_custom_config_dir/node.crt
xpack.security.transport.ssl.certificate_authorities: [ "/full_path_to_custom_config_dir/ca.root.crt" ]
xpack.security.transport.ssl.client_authentication: required

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /full_path_to_custom_config_dir/node_http.key
xpack.security.http.ssl.certificate: /full_path_to_custom_config_dir/node_http.crt
xpack.security.http.ssl.certificate_authorities: [ "/full_path_to_custom_config_dir/ca.root.crt" ]
xpack.security.http.ssl.client_authentication: required

zpraxis · July 17, 2019, 2:57pm

Both keys are encrypted and I have already created entries in the keystore with:

elasticsearch-keystore add xpack.security.transport.ssl.secure_key_passphrase
elasticsearch-keystore add xpack.security.http.ssl.secure_key_passphrase

I am able to see them with the list option.

zpraxis · July 18, 2019, 7:30pm

Exploring the code, specifically the class indicated in the stacktrace, it seems that transport properties are loaded fine, but HTTP ones are not. I can see the "xpack.http.ssl", but I think it refers to properties related to Watcher configuration.

Method loadSSLConfigurations, line 414:

/**
 * Parses the settings to load all SSLConfiguration objects that will be used.
 */
Map<SSLConfiguration, SSLContextHolder> loadSSLConfigurations() {
    Map<SSLConfiguration, SSLContextHolder> sslContextHolders = new HashMap<>();

    Map<String, Settings> sslSettingsMap = new HashMap<>();
    sslSettingsMap.put(XPackSettings.HTTP_SSL_PREFIX, getHttpTransportSSLSettings(settings));
    sslSettingsMap.put("xpack.http.ssl", settings.getByPrefix("xpack.http.ssl."));
    sslSettingsMap.putAll(getRealmsSSLSettings(settings));
    sslSettingsMap.putAll(getMonitoringExporterSettings(settings));

    sslSettingsMap.forEach((key, sslSettings) -> {
        final SSLConfiguration configuration = new SSLConfiguration(sslSettings);
        storeSslConfiguration(key, configuration);
        sslContextHolders.computeIfAbsent(configuration, this::createSslContext);
    });

    final Settings transportSSLSettings = settings.getByPrefix(XPackSettings.TRANSPORT_SSL_PREFIX);
    final SSLConfiguration transportSSLConfiguration = new SSLConfiguration(transportSSLSettings);
    this.transportSSLConfiguration.set(transportSSLConfiguration);
    storeSslConfiguration(XPackSettings.TRANSPORT_SSL_PREFIX, transportSSLConfiguration);
    Map<String, Settings> profileSettings = getTransportProfileSSLSettings(settings);
    sslContextHolders.computeIfAbsent(transportSSLConfiguration, this::createSslContext);
    profileSettings.forEach((key, profileSetting) -> {
        final SSLConfiguration configuration = new SSLConfiguration(profileSetting);
        storeSslConfiguration(key, configuration);
        sslContextHolders.computeIfAbsent(configuration, this::createSslContext);
    });

    return Collections.unmodifiableMap(sslContextHolders);
}

github.com

elastic/elasticsearch/blob/afcdbd2bc03a9df3b5a45b6f7dfcc2b5e52df404/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ssl/SSLService.java

/*
 * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
 * or more contributor license agreements. Licensed under the Elastic License;
 * you may not use this file except in compliance with the Elastic License.
 */
package org.elasticsearch.xpack.core.ssl;

import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.nio.conn.ssl.SSLIOSessionStrategy;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.lucene.util.SetOnce;
import org.elasticsearch.ElasticsearchException;
import org.elasticsearch.common.CheckedSupplier;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.env.Environment;
import org.elasticsearch.xpack.core.XPackSettings;
import org.elasticsearch.xpack.core.common.socket.SocketAccess;
import org.elasticsearch.xpack.core.ssl.cert.CertificateInfo;

This file has been truncated. show original

zpraxis · August 15, 2019, 7:09pm

After reviewing the code of the SSLService class, we realized that the following line is causing problems:

711: final SSLConfiguration configuration = sslConfigurations.get(contextName);

After some debugging, we conclude that maybe it is due to the fact that the object sslConfigurations is an unmodifiableMap. We printed out all the elements of the hash map, and the context name that Elasticsearch is looking up does exist in the map, but fails to retrieve it. This happens when we execute Eleasticsearch with the Oracle JVM that is installed in our server. If we run the Elasticsearch version that includes its own OpenDJK JVM, it works fine. The context that is not finding is the SSL context (the Transport context is retrieved correctly as far from what we see).

Link to the code: https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ssl/SSLService.java

system · September 12, 2019, 7:21pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.