Elasticsearch 7.2.1 startup failed load search guard plugin with java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSLPrivateKeyMethod

[Rachangouda]

Hi,

I have followed below steps to start elasticsearch 7.2.1 with search guard version-36.1.0 as mentioned in https://docs.search-guard.com/latest/search-guard-versions

1. created elasticsearch image from elasticsearch-7.x.repo yum repo with user as "elastic".
2. installed search guard plugin from https://oss.sonatype.org/service/local/repositories/releases/content/com/floragunn/search-guard-7/7.2.1-36.1.0/search-guard-7-7.2.1-36.1.0.zip
3. added openssl dependency for Search Guard
   openssl netty-tcnative-openssl-1.1.0j-static-2.0.20.Final-non-fedora-linux-x86_64.jar

When elasticsearch is trying to start
getting below error:

Caused by: java.lang.NoClassDefFoundError: io/netty/internal/tcnative/SSLPrivateKeyMethod

Full log:

[2019-09-03T06:30:31,664][WARN ][org.elasticsearch.bootstrap.ElasticsearchUncaughtExceptionHandler] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.2.1.jar:7.2.1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.2.1.jar:7.2.1]
Caused by: java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.2.1.jar:7.2.1]
.
.
Caused by: java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSLPrivateKeyMethod
at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_201]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_201]
at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:817) ~[?:1.8.0_201]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_201]
at io.netty.handler.ssl.SslContext.newServerContextInternal(SslContext.java:423) ~[?:?]
at io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:447) ~[?:?]
at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore$1.run(DefaultSearchGuardKeyStore.java:785) ~[?:?]
at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore$1.run(DefaultSearchGuardKeyStore.java:782) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_201]
at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.buildSSLContext0(DefaultSearchGuardKeyStore.java:782) ~[?:?]
at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.buildSSLServerContext(DefaultSearchGuardKeyStore.java:730) ~[?:?]
at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:315) ~[?:?]
at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.(DefaultSearchGuardKeyStore.java:151) ~[?:?]
at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.(SearchGuardSSLPlugin.java:194) ~[?:?]
at com.floragunn.searchguard.SearchGuardPlugin.(SearchGuardPlugin.java:211) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_201]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:163) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.node.Node.(Node.java:307) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.node.Node.(Node.java:251) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.2.1.jar:7.2.1]
... 6 more

After verifying the binaries
added netty-tcnative-2.0.25.Final.jar and removed the openssl netty-tcnative-openssl-1.1.0j-static-2.0.20.Final-non-fedora-linux-x86_64.jar

then elasticsearch started .
But search guard is not able to connect with elasticsearch. getting below error from search guard.
None of the configured nodes are available

Does Elasticsearch is having problem or Search guard is having problem?

[Rachangouda]

Please update anyone faced same problem.

[polyfractal (Zachary Tong)]

SearchGuard is not our product, so we don't have any insight into how it works or what might be wrong. I would suggest asking the SearchGuard folks what is wrong with their plugin.