Logstash Error after updating form 7.8.0 to 7.10.0, Pipeline error {:pipeline_id=>"main", :exception=> javax.net.ssl.SSLException: failed to initialize the server-side SSL context

Elastic Stack Logstash
1

I update ELK stack from 7.8.0 to 7.10.0.
I am facing this Error in logstash:

[ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>
javax.net.ssl.SSLException: failed to initialize the server-side SSL context, 
:backtrace=>[
"io.netty.handler.ssl.JdkSslServerContext.newSSLContext(io/netty/handler/ssl/JdkSslServerContext.java:288)", 
"io.netty.handler.ssl.JdkSslServerContext.<init>(io/netty/handler/ssl/JdkSslServerContext.java:247)", 
"io.netty.handler.ssl.SslContext.newServerContextInternal(io/netty/handler/ssl/SslContext.java:465)", 
"io.netty.handler.ssl.SslContextBuilder.build(io/netty/handler/ssl/SslContextBuilder.java:571)", 
"jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", 
"jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)", 
"jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", 
"java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)", 
"org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:426)", 
"org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:293)", 
"usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_0_dot_6_minus_java.lib.logstash.inputs.tcp.compat_ssl_options.toSslContext(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.0.6-java/lib/logstash/inputs/tcp/compat_ssl_options.rb:127)", 
"usr.share.logstash.vendor.bundle.jruby2_dot_5_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_0_dot_6_minus_java.lib.logstash.inputs.tcp.compat_ssl_options.RUBY$method$toSslContext$0$__VARARGS__(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_input_minus_tcp_minus_6_dot_0_dot_6_minus_java/lib/logstash/inputs/tcp//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.0.6-java/lib/logstash/inputs/tcp/compat_ssl_options.rb)", 
"usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_0_dot_6_minus_java.lib.logstash.inputs.tcp.get_ssl_context(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.0.6-java/lib/logstash/inputs/tcp.rb:369)", 
"usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_0_dot_6_minus_java.lib.logstash.inputs.tcp.RUBY$method$get_ssl_context$0$__VARARGS__(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_input_minus_tcp_minus_6_dot_0_dot_6_minus_java/lib/logstash/inputs//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.0.6-java/lib/logstash/inputs/tcp.rb)", 
"usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_0_dot_6_minus_java.lib.logstash.inputs.tcp.register(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.0.6-java/lib/logstash/inputs/tcp.rb:145)", 
"usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_tcp_minus_6_dot_0_dot_6_minus_java.lib.logstash.inputs.tcp.RUBY$method$register$0$__VARARGS__(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_input_minus_tcp_minus_6_dot_0_dot_6_minus_java/lib/logstash/inputs//usr/share/logstash/vendor/bundle//2.5.0/gems/logstash-input-tcp-6.0.6-java/lib/logstash/inputs/tcp.rb)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1809)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$register_plugins$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_inputs(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:386)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$start_inputs$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:311)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$start_workers$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.run(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$run$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash//usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137)", 
"org.jruby.RubyProc.call(org/jruby/RubyProc.java:318)", 
"java.lang.Thread.run(java/lang/Thread.java:834)"], 
"pipeline.sources"=>["/etc/logstash/conf.d/pipeline.conf"], 
:thread=>"#<Thread:0x18f5cec9 run>"}

My pipeline.conf looks like :

input {
    udp {
        port => 5000
        type => syslog
    }
    udp {
        port => 5001
        type => json
    }
    tcp {
        port => 5001
        type => json
        ssl_enable => true
        ssl_key => "....."
        ssl_cert => "...."
        ssl_extra_chain_certs => ["...."]
        ssl_verify => false
        add_field => {"ssl" => "on"}
    }
}

## Add your filters / logstash plugins configuration here

filter {
    mutate {
      remove_field => [ "port" ]
    }

    if [type] == "syslog" {
        grok {
            match => { "message" => "\A%{TIMESTAMP_ISO8601:tmptimestamp}\|%{HOSTNAME:cluster}\|%{HOSTNAME:hostname}\|%{HOSTNAME:app}\|%{GREEDYDATA:message}\Z" }
            overwrite => [ "message" ]
        }
        date {
            match => [ "tmptimestamp", "ISO8601" ]
            remove_field => [ "tmptimestamp" ]
        }
    }

    mutate {
       add_field => { "token" => "JnQKKkJNnTmNRpuremJWXQMLFQAVKlwh" }
    }
}
    mutate {
        copy => { "_id" => "[@metadata][_id]" }
        remove_field => ["_id"]
    }
    mutate {
        remove_field =>  ["host"]
        remove_field => ["CancelPenalty","Search","Offer","Rateplan","Success","PropertyChangedMessage","Person","day","Rate","SearchRequest","ExternalRatesUpdate","SalesforceOpportunity","Tax"]
    }

}

output {

    if [type] == "syslog" and "_grokparsefailure" in [tags] {
        file { path => "~/log/failed_syslog_events-%{+YYYY-MM-dd}" }
    }

    elasticsearch {
        hosts => "localhost:9200"
    }
}
2

Same issue (7.6 w/ Adoptjdk 11.0.6 to 7.10 w/ bundled 11.0.8)

javax.net.ssl.SSLException: failed to initialize the server-side SSL context, :backtrace=>["io.netty.handler.ssl.JdkSslServerContext.newSSLContext(io/netty/handler/ssl/JdkSslServerContext.java:288)"...

Pipeline:

input {
tcp {
    port => 6514
    ssl_enable => true
    ssl_cert => "E:/logstash/config/certs/servername.pem"
    ssl_key => "E:/logstash/config/certs/servername.key"
    ssl_certificate_authorities => "E:/logstash/config/certs/ca_chain.pem"
    ssl_key_passphrase => "somepassword"
    ssl_verify => false
}
}

Also having java issues with elasticsearch in 7.10, seems to be alot of issues with the jdk switch.

Note - I tried using the same JDK I was using in 7.6 and the SSL issues persist, so not an issue with the JDK it seems.

3

I have an open case for these SSL issues but have not gotten responses on those either :disappointed_relieved:

4

I'm having the same issue.

OS: Ubuntu 18.04.5 LTS
Java: openjdk-8
Certificates are from: InCommon

Did some quick testing, and it doesn't seem to matter if the input is Beats, tcp or http.
Removing any TLS related configuration allows logstash to start.

Switched to openjdk 14 - no luck.

Converted enough of my config to run in a docker container and was able to reproduce the problem there as well.

6

I don't know if the JVM options file in Elasticsearch has something to do with the SSLException logstash error. But after updating Elasticsearch didn't start directly. I had to change the JVM options file in /etc/elasticsearch/jvm.options to look like:

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms1g
-Xmx1g

################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don't tamper with them unless
## you understand what you are doing
##
################################################################

## GC configuration
8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly

## G1GC Configuration
# NOTE: G1 GC is only supported on JDK version 10 or later
# to use G1GC, uncomment the next two lines and update the version on the
# following three lines to your version of the JDK
# 10-13:-XX:-UseConcMarkSweepGC
# 10-13:-XX:-UseCMSInitiatingOccupancyOnly
14-:-XX:+UseG1GC
14-:-XX:G1ReservePercent=25
14-:-XX:InitiatingHeapOccupancyPercent=30

## JVM temporary directory
-Djava.io.tmpdir=${ES_TMPDIR}

## heap dumps

# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError

# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=/var/lib/elasticsearch

# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log

## JDK 8 GC logging
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/elasticsearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m
8:-XX:GCLogFileSize=64m

# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m
7

I tested this on a different logstash server and didn't have the same problem.

8

I believe the jvm options is a separate issue. When I upgraded to 7.10 I had to revert my custom config to the default included with 7.10 since it doesn't seem to account for the change in jdk.

9

Anyone with a solution for this problem ????

10

Support asked me to try adding this via jvm.options.d

-Djdk.security.allowNonCaAnchor=true

Im OOO currently so you can give it a go.

11

@jaredo thanks for passing the solution.

Unfortunately, didn't help out. I also guess that you mean to add the line -Djdk.security.allowNonCaAnchor=true to jvm.options not into jvm.options.d in /etc/logstash there are no such file jvm.options.d

So i found the following Worning when i run the command
/usr/share/logstash/bin# ./logstash-plugin --help

Using bundled JDK: /usr/share/logstash/jdk
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/jopenssl.jar) to field java.security.MessageDigest.provider
WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

Could this has something to do with the error ?? Is the jkd version AdoptOpenJDK 11.0.8 the sourse of this error ??

12

Yea I believe that is a JDK15 setting. I am having issues in Elastic + Logstash, that was a recommendation for ES on JDK15. Not sure about logstash was hoping it would be similar.

13

I do not think so. Java is allowing the reflective access, but warning you that a future version will not.

14

So no solution or workaround this error ??

15

It is not an error. It is a warning that something is going to change in a future release.

16

@Badger I am asking about the error in the top of the page

17

I solve this problem by removing the removing the ssl_extra_chain_certs . For explanation, I think you might be hitting this: #160

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.