Kubernetes deployment - 3rd Stateful set replica always fail with java.lang.ClassNotFoundException: com.fasterxml.jackson.core.io.JsonStringEncoder

[kolslearningid]

I am trying to build a three replica stateful set Elasticsearch cluster in EKS using helm chart. When I deploy with one or two replicas the PODs are getting created and attached in the cluster mode. When I increase the replica count to 3 then the new pod always gets fails with the below error during startup. I am trying to create all three stateful set POD's using role MASTER+DATA. Wondering how come other two POD's using the same image got started successfully whereas the 3rd one always ending up with a class not found issue.

Docker image : elasticsearch:8.7.1

Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.core.io.JsonStringEncoder

Getting below error when I try to use docker image : docker.elastic.co/elasticsearch/elasticsearch:7.17.9

whatever the image I use, the 3rd stateful set POD always gets stuck in some class not found issue.

ElasticsearchException[Could not find plugin class [org.elasticsearch.xpack.security.Security]]; nested: ClassNotFoundException[org.elasticsearch.xpack.security.Security];
Likely root cause: java.lang.ClassNotFoundException: org.elasticsearch.xpack.security.Security
    at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
    at java.base/java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:872)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:495)
    at java.base/java.lang.Class.forName(Class.java:474)
    at org.elasticsearch.plugins.PluginsService.loadPluginClass(PluginsService.java:777)
    at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:738)
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:533)
    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:170)
    at org.elasticsearch.node.Node.<init>(Node.java:411)
    at org.elasticsearch.node.Node.<init>(Node.java:309)
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)
    at org.elasticsearch.cli.Command.main(Command.java:77)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)
For complete error details, refer to the log at /usr/share/elasticsearch/logs/elasticsearch.log
2023-05-23 19:03:59,695328 UTC [432] INFO  Main.cc@111 Parent process died - ML controller exiting