Elasticsearch will not start after installing 7.9.2

Elastic Stack Elasticsearch
21 
    at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:530) ~[elasticsearch-6.8.7.jar:6.8.7                           ]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.performAction(TransportReplica                           tionAction.java:881) ~[elasticsearch-6.8.7.jar:6.8.7]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.performLocalAction(TransportRe                           plicationAction.java:832) ~[elasticsearch-6.8.7.jar:6.8.7]
    at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationActi                           on.java:819) ~[elasticsearch-6.8.7.jar:6.8.7]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.8.7.jar:6.                           8.7]
    at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:1                           72) ~[elasticsearch-6.8.7.jar:6.8.7]
    at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:1                           00) ~[elasticsearch-6.8.7.jar:6.8.7]
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) ~[elasticsear                           ch-6.8.7.jar:6.8.7]
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:84) ~[                           ?:?]
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.7.jar:6.8.7]
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.                           java:170) ~[?:?]
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.7.jar:6.8.7]
    at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:377) ~[?:                           ?]
    at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:186) ~[?:?]
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:170)                            ~[?:?]
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.jav                           a:156) ~[?:?]
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.7.jar:6.8.7]
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(Authenticatio                           nService.java:246) ~[?:?]
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(A                           uthenticationService.java:306) ~[?:?]
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(Authentica                           tionService.java:317) ~[?:?]
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.                           java:244) ~[?:?]
    at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:19                           6) ~[?:?]
    at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139) ~[?:?]
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:153) ~[                           ?:?]
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$1(SecurityActionFilter.java:90) ~[                           ?:?]
    at org.elasticsearch.xpack.core.security.SecurityContext.executeAsUser(SecurityContext.java:96) ~[?:?]
    at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:88) ~[?:?]
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165) ~[elasticsear                           ch-6.8.7.jar:6.8.7]
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) ~[elasticsearch-6.8.7.jar:6.8.7]
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81) ~[elasticsearch-6.8.7.jar:6.8.7]
    at org.elasticsearch.index.seqno.GlobalCheckpointSyncAction.updateGlobalCheckpointForShard(GlobalCheckpointSyncAction.                           java:90) ~[elasticsearch-6.8.7.jar:6.8.7]
    at org.elasticsearch.index.IndexService.lambda$createShard$5(IndexService.java:408) ~[elasticsearch-6.8.7.jar:6.8.7]
    at org.elasticsearch.index.shard.IndexShard.maybeSyncGlobalCheckpoint(IndexShard.java:2204) ~[elasticsearch-6.8.7.jar:                           6.8.7]
    at org.elasticsearch.index.IndexService.lambda$maybeSyncGlobalCheckpoints$11(IndexService.java:777) ~[elasticsearch-6.                           8.7.jar:6.8.7]
    at org.elasticsearch.index.IndexService.lambda$sync$12(IndexService.java:800) ~[elasticsearch-6.8.7.jar:6.8.7]
    at org.elasticsearch.index.shard.IndexShard.lambda$runUnderPrimaryPermit$17(IndexShard.java:2635) ~[elasticsearch-6.8.                           7.jar:6.8.7]
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.7.jar:6.8.7]
    at org.elasticsearch.index.shard.IndexShard.lambda$wrapPrimaryOperationPermitListener$14(IndexShard.java:2588) ~[elast                           icsearch-6.8.7.jar:6.8.7]
    at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.7.jar:6.8.7]
    at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273) ~[elasticsear                           ch-6.8.7.jar:6.8.7]
    at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240) ~[elasticsear                           ch-6.8.7.jar:6.8.7]
    at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2563) ~[elasticsearch-6.8.7.                           jar:6.8.7]
    at org.elasticsearch.index.shard.IndexShard.runUnderPrimaryPermit(IndexShard.java:2639) ~[elasticsearch-6.8.7.jar:6.8.                           7]
    at org.elasticsearch.index.IndexService.sync(IndexService.java:799) ~[elasticsearch-6.8.7.jar:6.8.7]
    at org.elasticsearch.index.IndexService.maybeSyncGlobalCheckpoints(IndexService.java:777) ~[elasticsearch-6.8.7.jar:6.                           8.7]
    at org.elasticsearch.index.IndexService.access$700(IndexService.java:100) ~[elasticsearch-6.8.7.jar:6.8.7]
    at org.elasticsearch.index.IndexService$AsyncGlobalCheckpointTask.runInternal(IndexService.java:938) ~[elasticsearch-6                           .8.7.jar:6.8.7]
    at org.elasticsearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:144) ~[elasticsearch-6.8.7.ja                           r:6.8.7]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[elas                           ticsearch-6.8.7.jar:6.8.7]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]

Caused by: org.elasticsearch.transport.TransportException: TransportService is closed stopped can't send request
at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:626) ~[elasticsearch-6.8.7.j ar:6.8.7]
... 56 more
[2021-02-05T04:36:09,897][INFO ][o.e.n.Node ] [dev-es-00] stopped
[2021-02-05T04:36:09,897][INFO ][o.e.n.Node ] [dev-es-00] closing ...
[2021-02-05T04:36:09,914][INFO ][o.e.n.Node ] [dev-es-00] closed

indent preformatted text by 4 spaces
22

@mike_wills

Those logs look from Feb 5th... long ago...

Did you try to start elasticsearch in the foreground as I suggested?

ES_PATH_CONF=/etc/elasticsearch /usr/share/elasticsearch/bin/elasticsearch

Not sure what any of us can do until we see logs from the new install...

23

I grabbed the wrong logs. Yes, I tried it in the foreground and it still wouldn't start up.

java.lang.RuntimeException: can not run elasticsearch as root
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:111) ~[elasticsearch-7.9.2.jar:7.9.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178) ~[elasticsearch-7.9.2.jar:7.9.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.9.2.jar:7.9.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.9.2.jar:7.9.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.9.2.jar:7.9.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.9.2.jar:7.9.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.9.2.jar:7.9.2]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.9.2.jar:7.9.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.9.2.jar:7.9.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.9.2.jar:7.9.2]
[2021-03-02T00:34:56,215][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [dev-es-00.autumnal.local] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.9.2.jar:7.9.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.9.2.jar:7.9.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.9.2.jar:7.9.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.9.2.jar:7.9.2]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.9.2.jar:7.9.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.9.2.jar:7.9.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.9.2.jar:7.9.2]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:111) ~[elasticsearch-7.9.2.jar:7.9.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178) ~[elasticsearch-7.9.2.jar:7.9.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.2.jar:7.9.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.2.jar:7.9.2]
        ... 6 more
24

Yeah you can't run elasticsearch as root you would need to run as the correct user, and that might take more work

Let's try something different I did a little reading on journalctl this simple cheat sheet here.

Why not open 2 windows...

in the first run... let it come to and end leave it running this will tail the elasticsearch journal and should capture the startup exceptions.

sudo journalctl -ef -u elasticsearch

In the 2nd window run

sudo systemctl start elasticsearch

See if you see anything interesting in the first window, it should capture start-up errors if so paste in here.

25 
 18:00:12 dev-es-00.autumnal.local systemd[1]: Starting Elasticsearch...
Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: 2021-03-02 18:00:15,481 main ERROR RollingFileManager (/va                           r/log/elasticsearch/gc-gc_server.json) java.io.FileNotFoundException: /var/log/elasticsearch/gc-gc_server.json (Permission den                           ied) java.io.FileNotFoundException: /var/log/elasticsearch/gc-gc_server.json (Permission denied)
Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at java.base/java.io.FileOutputStream.open0(Native Method)
Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at java.base/java.io.FileOutputStream.open(FileOutputStrea                           m.java:291)
Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at java.base/java.io.FileOutputStream.<init>(FileOutputStr                           eam.java:234)
Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at java.base/java.io.FileOutputStream.<init>(FileOutputStr                           eam.java:155)
Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.appender.rolling.RollingF                           ileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:640)
Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.appender.rolling.RollingF                           ileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:608)
Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.appender.AbstractManager.                           getManager(AbstractManager.java:113)
Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.appender.OutputStreamMana                           ger.getManager(OutputStreamManager.java:114)
Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.appender.rolling.RollingF                           ileManager.getFileManager(RollingFileManager.java:188)
Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.appender.RollingFileAppen                           der$Builder.build(RollingFileAppender.java:145)
Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.appender.RollingFileAppen                           der$Builder.build(RollingFileAppender.java:61)
Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.config.plugins.util.Plugi                           nBuilder.build(PluginBuilder.java:123)
26 
 Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.config.AbstractConfigurat                           ion.createPluginObject(AbstractConfiguration.java:959)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.config.AbstractConfigurat                           ion.createConfiguration(AbstractConfiguration.java:899)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.config.AbstractConfigurat                           ion.createConfiguration(AbstractConfiguration.java:891)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.config.AbstractConfigurat                           ion.doConfigure(AbstractConfiguration.java:514)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.config.AbstractConfigurat                           ion.initialize(AbstractConfiguration.java:238)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.config.AbstractConfigurat                           ion.start(AbstractConfiguration.java:250)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.LoggerContext.setConfigur                           ation(LoggerContext.java:547)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.LoggerContext.start(Logge                           rContext.java:263)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.common.logging.LogConfigurator.config                           ure(LogConfigurator.java:236)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.common.logging.LogConfigurator.config                           ure(LogConfigurator.java:129)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.ja                           va:354)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elastics                           earch.java:170)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elast                           icsearch.java:161)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(E                           nvironmentAwareCommand.java:86)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(                           Command.java:127)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.cli.Command.main(Command.java:90)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elastics                           earch.java:126)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elastics                           earch.java:92)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: 2021-03-02 18:00:15,484 main ERROR Could not create plugin                            of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateExcep                           tion: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@13d4992d] un                           able to create manager for [/var/log/elasticsearch/gc-gc_server.json] with data [org.apache.logging.log4j.core.appender.rollin                           g.RollingFileManager$FactoryData@302f7971[pattern=/var/log/elasticsearch/gc-gc-%d{yyyy-MM-dd}-%i.json.gz, append=true, buffere                           dIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval                           =1, modulate=true), SizeBasedTriggeringPolicy(size=134217728)]), strategy=DefaultRolloverStrategy(min=-2147483648, max=2147483                           647, useMax=false), advertiseURI=null, layout=ESJsonLayout{patternLayout={"type": "server", "timestamp": "%d{yyyy-MM-dd'T'HH:m                           m:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "%node_name",                            "message": "%notEmpty{%enc{%marker}{JSON} }%enc{%.-10000m}{JSON}"%notEmpty{, %node_and_cluster_id }%exceptionAsJson }
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: }, filePermissions=null, fileOwner=null]] java.lang.Illega                           lStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@1                           3d4992d] unable to create manager for [/var/log/elasticsearch/gc-gc_server.json] with data [org.apache.logging.log4j.core.appe                           nder.rolling.RollingFileManager$FactoryData@302f7971[pattern=/var/log/elasticsearch/gc-gc-%d{yyyy-MM-dd}-%i.json.gz, append=tr                           ue, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=                           0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=134217728)]), strategy=DefaultRolloverStrategy(min=-2147483648,                            max=2147483647, useMax=false), advertiseURI=null, layout=ESJsonLayout{patternLayout={"type": "server", "timestamp": "%d{yyyy-M                           M-dd'T'HH:mm:ss,SSSZZ}", "level": "%p", "component": "%c{1.}", "cluster.name": "${sys:es.logs.cluster_name}", "node.name": "%n                           ode_name", "message": "%notEmpty{%enc{%marker}{JSON} }%enc{%.-10000m}{JSON}"%notEmpty{, %node_and_cluster_id }%exceptionAsJson                            }
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: }, filePermissions=null, fileOwner=null]]
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.appender.AbstractManager.                           getManager(AbstractManager.java:115)
27 
   Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.appender.OutputStreamMana                           ger.getManager(OutputStreamManager.java:114)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.appender.rolling.RollingF                           ileManager.getFileManager(RollingFileManager.java:188)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.appender.RollingFileAppen                           der$Builder.build(RollingFileAppender.java:145)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.appender.RollingFileAppen                           der$Builder.build(RollingFileAppender.java:61)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.config.plugins.util.Plugi                           nBuilder.build(PluginBuilder.java:123)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.config.AbstractConfigurat                           ion.createPluginObject(AbstractConfiguration.java:959)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.config.AbstractConfigurat                           ion.createConfiguration(AbstractConfiguration.java:899)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.config.AbstractConfigurat                           ion.createConfiguration(AbstractConfiguration.java:891)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.config.AbstractConfigurat                           ion.doConfigure(AbstractConfiguration.java:514)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.config.AbstractConfigurat                           ion.initialize(AbstractConfiguration.java:238)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.config.AbstractConfigurat                           ion.start(AbstractConfiguration.java:250)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.LoggerContext.setConfigur                           ation(LoggerContext.java:547)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.LoggerContext.start(Logge                           rContext.java:263)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.common.logging.LogConfigurator.config                           ure(LogConfigurator.java:236)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.common.logging.LogConfigurator.config                           ure(LogConfigurator.java:129)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.ja                           va:354)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elastics                           earch.java:170)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elast                           icsearch.java:161)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(E                           nvironmentAwareCommand.java:86)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(                           Command.java:127)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.cli.Command.main(Command.java:90)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elastics                           earch.java:126)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elastics                           earch.java:92)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: 2021-03-02 18:00:15,485 main ERROR Unable to invoke factor                           y method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateEx                           ception: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateEx                           ception: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.config.plugins.util.Plugi                           nBuilder.findFactoryMethod(PluginBuilder.java:235)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.config.plugins.util.Plugi                           nBuilder.build(PluginBuilder.java:135)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.config.AbstractConfigurat                           ion.createPluginObject(AbstractConfiguration.java:959)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.config.AbstractConfigurat                           ion.createConfiguration(AbstractConfiguration.java:899)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.config.AbstractConfigurat                           ion.createConfiguration(AbstractConfiguration.java:891)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.config.AbstractConfigurat                           ion.doConfigure(AbstractConfiguration.java:514)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.config.AbstractConfigurat                           ion.initialize(AbstractConfiguration.java:238)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.config.AbstractConfigurat                           ion.start(AbstractConfiguration.java:250)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.LoggerContext.setConfigur                           ation(LoggerContext.java:547)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.apache.logging.log4j.core.LoggerContext.start(Logge                           rContext.java:263)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.common.logging.LogConfigurator.config                           ure(LogConfigurator.java:236)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.common.logging.LogConfigurator.config                           ure(LogConfigurator.java:129)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.ja                           va:354)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elastics                           earch.java:170)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elast                           icsearch.java:161)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(E                           nvironmentAwareCommand.java:86)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(                           Command.java:127)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.cli.Command.main(Command.java:90)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elastics                           earch.java:126)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elastics                           earch.java:92)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: 2021-03-02 18:00:15,551 main ERROR RollingFileManager (/va                           r/log/elasticsearch/gc-gc.log) java.io.FileNotFoundException: /var/log/elasticsearch/gc-gc.log (Permission denied) java.io.Fil                           eNotFoundException: /var/log/elasticsearch/gc-gc.log (Permission denied)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at java.base/java.io.FileOutputStream.open0(Native Method)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at java.base/java.io.FileOutputStream.open(FileOutputStrea                           m.java:291)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at java.base/java.io.FileOutputStream.<init>(FileOutputStr                           eam.java:234)
    Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: at java.base/java.io.FileOutputStream.<init>(FileOutputStr                           eam.java:155)
28 
   Mar 02 18:00:15 dev-es-00.autumnal.local systemd-entrypoint[15653]: 2021-03-02 18:00:15,571 main ERROR Unable to locate append                           er "rolling_old" for logger config "root"
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: uncaught exception in thread [main]
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: java.lang.IllegalStateException: failed to load plugin cla                           ss [org.elasticsearch.xpack.security.Security]
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: Likely root cause: java.lang.IllegalArgumentException: Inc                           orrect realm settings found. Realm settings have been changed to include the type as part of the setting key.
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: For example 'xpack.security.authc.realms.file.my_file.orde                           r'
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: Found invalid config: xpack.security.authc.realms.active_d                           irectory.secure_bind_password
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: Please see the breaking changes documentation.
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.xpack.security.Security.validateRealm                           Settings(Security.java:925)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.xpack.security.Security.runStartupChe                           cks(Security.java:343)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.xpack.security.Security.<init>(Securi                           ty.java:330)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.xpack.security.Security.<init>(Securi                           ty.java:320)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at java.base/jdk.internal.reflect.NativeConstructorAccesso                           rImpl.newInstance0(Native Method)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at java.base/jdk.internal.reflect.NativeConstructorAccesso                           rImpl.newInstance(NativeConstructorAccessorImpl.java:64)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at java.base/jdk.internal.reflect.DelegatingConstructorAcc                           essorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at java.base/java.lang.reflect.Constructor.newInstanceWith                           Caller(Constructor.java:500)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at java.base/java.lang.reflect.Constructor.newInstance(Con                           structor.java:481)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.plugins.PluginsService.loadPlugin(Plu                           ginsService.java:691)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.plugins.PluginsService.loadBundle(Plu                           ginsService.java:642)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.plugins.PluginsService.loadBundles(Pl                           uginsService.java:473)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.plugins.PluginsService.<init>(Plugins                           Service.java:165)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.node.Node.<init>(Node.java:328)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.node.Node.<init>(Node.java:277)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstra                           p.java:227)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.j                           ava:227)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.ja                           va:393)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elastics                           earch.java:170)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elast                           icsearch.java:161)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(E                           nvironmentAwareCommand.java:86)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(                           Command.java:127)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.cli.Command.main(Command.java:90)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elastics                           earch.java:126)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elastics                           earch.java:92)
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: For complete error details, refer to the log at /var/log/e                           lasticsearch/gc-gc.log
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd[1]: Failed to start Elasticsearch.
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd[1]: Unit elasticsearch.service entered failed state.
    Mar 02 18:00:17 dev-es-00.autumnal.local systemd[1]: elasticsearch.service failed.
29

Ok we are getting somewhere...

So we can easily see a couple things, at this point you have decent logs, read them and start to address the errors.

/var/log/elasticsearch/gc-gc_server.json) java.io.FileNotFoundException: /var/log/elasticsearch/gc-gc_server.json (Permission den

So make sure there is a directory /var/log/elasticsearch and it is writable

2nd you are going to need to update you elasticsearch.yml to 7.9.3 specs, you have some invalid security settings.

Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.security.Security]
Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: Likely root cause: java.lang.IllegalArgumentException: Incorrect realm settings found. Realm settings have been changed to include the type as part of the setting key.
Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: For example 'xpack.security.authc.realms.file.my_file.order'
Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: Found invalid config: xpack.security.authc.realms.active_directory.secure_bind_password
Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: Please see the breaking changes documentation.
Mar 02 18:00:17 dev-es-00.autumnal.local systemd-entrypoint[15653]: at org.elasticsearch.xpack.security.Security.validateRealmSettings(Security.java:925)

So this part of your elasticsearch.yml looks like it needs to be fixed

xpack:
    security:
        authc:
            realms:
                native1:
                    type: native
                    order: 0
                active_directory:
                    type: active_directory
                    order: 1
                    domain_name: 
                    url: 
                    bind_dn: 
                    load_balance:
                        type: "round_robin"
                    unmapped_groups_as_roles: true
                file1:
                    type: file
                    order: 0

per the error message

"Realm settings have been changed to include the type as part of the setting key."
So you need to go through and update that whole security config.

Looks like 6.8

xpack:
  security:
    authc:
      realms:
        file1:
          type: file
          order: 0

Looks like 7.9

xpack:
  security:
    authc:
      realms:
        file:
          file1:
            order: 0

So keep fixing, running and looking at the logs... you should be able to figure it out from here...

30

okay thanks

31

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.