Logstash and Elasticsearch using HTTPS

Hi!

I'm having trouble using HTTPS between Logstash and Elasticsearch. I have already enabled xpack.security features via:

xpack.security.enabled: true

And added the corresponding *.transport.* and *.ssl.* properties in elasticsearch.yml.

Also, in Logstash, I have:

output {
    elasticsearch {
      hosts => [ "my_ip_adress:9200" ]
      keystore => "${HOME}/elastic-ops/logstash/config/logstash.p12"
      keystore_password => "${keystore_password}"
      ssl => true
      ssl_certificate_verification => true
      cacert => "${HOME}/elastic-ops/logstash/config/ca.root.crt"
  }
}

For some reason, when I start both of them, I get the following in Logstash log:

[2019-07-16T19:44:03,829][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://my_ip_address:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://my_ip_address:9200/][Manticore::SocketException] Connection reset"}

And in Elasticsearch log I get something like the following:

java.lang.IllegalArgumentException: did not find a SSLContext for [SSLConfiguration{keyConfig=[keyPath=[/home/d_elastic/elastic-ops/elasticsearch/config/node_http.key], certPaths=[/home/d_elastic/elastic-ops/elasticsearch/config/node_http.crt]], trustConfig=ca=[/home/d_elastic/elastic-ops/elasticsearch/config/ca.root.crt]], cipherSuites=[[TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA]], supportedProtocols=[[TLSv1.2, TLSv1.1]], sslClientAuth=[REQUIRED], verificationMode=[FULL]}]
at org.elasticsearch.xpack.core.ssl.SSLService.sslContextHolder(SSLService.java:304) ~[x-pack-core-7.2.0.jar:7.2.0]
at org.elasticsearch.xpack.core.ssl.SSLService.sslContext(SSLService.java:292) ~[x-pack-core-7.2.0.jar:7.2.0]
at org.elasticsearch.xpack.core.ssl.SSLService.createSSLEngine(SSLService.java:248) ~[x-pack-core-7.2.0.jar:7.2.0]
at org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport$HttpSslChannelHandler.initChannel(SecurityNetty4HttpServerTransport.java:81) ~[?:?]
at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:969) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:610) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1461) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1126) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:651) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:515) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:428) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:487) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.35.Final.jar:4.1.35.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:405) [netty-common-4.1.35.Final.jar:4.1.35.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [netty-transport-4.1.35.Final.jar:4.1.35.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.35.Final.jar:4.1.35.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.35.Final.jar:4.1.35.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

This same stacktrace I get when I try to run for the first time the tool to set the passwords for the built-in users.

What could be happening? Am I missing something?

Can you post your elasticsearch.yml please?

It looks like there's something incorrect in your config, and we're failing in a very unhelpful way.

Thanks for your prompt response!

Aside from basic properties set like paths, the rest of the file is like the following:

cluster.name: my_cluster
node.name: ${HOSTNAME}-node

path.data: /full_path_to_custom_data_dir/data/
path.logs: /full_path_to_custom_logs_dir/logs/

network.host: my_ip
http.port: 9200

discovery.seed_hosts: ["my_ip:9200"]
cluster.initial_master_nodes: ["${HOSTNAME}-node"]

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.key: /full_path_to_custom_config_dir/node.key
xpack.security.transport.ssl.certificate: /full_path_to_custom_config_dir/node.crt
xpack.security.transport.ssl.certificate_authorities: [ "/full_path_to_custom_config_dir/ca.root.crt" ]
xpack.security.transport.ssl.client_authentication: required

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /full_path_to_custom_config_dir/node_http.key
xpack.security.http.ssl.certificate: /full_path_to_custom_config_dir/node_http.crt
xpack.security.http.ssl.certificate_authorities: [ "/full_path_to_custom_config_dir/ca.root.crt" ]
xpack.security.http.ssl.client_authentication: required

Both keys are encrypted and I have already created entries in the keystore with:

elasticsearch-keystore add xpack.security.transport.ssl.secure_key_passphrase
elasticsearch-keystore add xpack.security.http.ssl.secure_key_passphrase

I am able to see them with the list option.

Exploring the code, specifically the class indicated in the stacktrace, it seems that transport properties are loaded fine, but HTTP ones are not. I can see the "xpack.http.ssl", but I think it refers to properties related to Watcher configuration.

Method loadSSLConfigurations, line 414:

/**
 * Parses the settings to load all SSLConfiguration objects that will be used.
 */
Map<SSLConfiguration, SSLContextHolder> loadSSLConfigurations() {
    Map<SSLConfiguration, SSLContextHolder> sslContextHolders = new HashMap<>();

    Map<String, Settings> sslSettingsMap = new HashMap<>();
    sslSettingsMap.put(XPackSettings.HTTP_SSL_PREFIX, getHttpTransportSSLSettings(settings));
    sslSettingsMap.put("xpack.http.ssl", settings.getByPrefix("xpack.http.ssl."));
    sslSettingsMap.putAll(getRealmsSSLSettings(settings));
    sslSettingsMap.putAll(getMonitoringExporterSettings(settings));

    sslSettingsMap.forEach((key, sslSettings) -> {
        final SSLConfiguration configuration = new SSLConfiguration(sslSettings);
        storeSslConfiguration(key, configuration);
        sslContextHolders.computeIfAbsent(configuration, this::createSslContext);
    });

    final Settings transportSSLSettings = settings.getByPrefix(XPackSettings.TRANSPORT_SSL_PREFIX);
    final SSLConfiguration transportSSLConfiguration = new SSLConfiguration(transportSSLSettings);
    this.transportSSLConfiguration.set(transportSSLConfiguration);
    storeSslConfiguration(XPackSettings.TRANSPORT_SSL_PREFIX, transportSSLConfiguration);
    Map<String, Settings> profileSettings = getTransportProfileSSLSettings(settings);
    sslContextHolders.computeIfAbsent(transportSSLConfiguration, this::createSslContext);
    profileSettings.forEach((key, profileSetting) -> {
        final SSLConfiguration configuration = new SSLConfiguration(profileSetting);
        storeSslConfiguration(key, configuration);
        sslContextHolders.computeIfAbsent(configuration, this::createSslContext);
    });

    return Collections.unmodifiableMap(sslContextHolders);
}

After reviewing the code of the SSLService class, we realized that the following line is causing problems:

711: final SSLConfiguration configuration = sslConfigurations.get(contextName);

After some debugging, we conclude that maybe it is due to the fact that the object sslConfigurations is an unmodifiableMap. We printed out all the elements of the hash map, and the context name that Elasticsearch is looking up does exist in the map, but fails to retrieve it. This happens when we execute Eleasticsearch with the Oracle JVM that is installed in our server. If we run the Elasticsearch version that includes its own OpenDJK JVM, it works fine. The context that is not finding is the SSL context (the Transport context is retrieved correctly as far from what we see).

Link to the code: https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ssl/SSLService.java

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.