Failed to start LSB, and Elasticsearch startup script does not exist or is not executable

PSandusky · February 24, 2021, 2:06pm

I've been having a rough time getting Elasticsearch to start at all. On the most recent attempt, I got a response that the startup script did not exist or was not executable. When I pulled systemctl status and then journalctl -xe, I received the following:

    selks-user@SELKS:~$ sudo systemctl status elasticsearch.service
    ● elasticsearch.service - LSB: Starts elasticsearch
       Loaded: loaded (/etc/init.d/elasticsearch; generated)
       Active: failed (Result: exit-code) since Wed 2021-02-24 08:36:43 EST; 49s ago
         Docs: man:systemd-sysv-generator(8)
      Process: 3606 ExecStart=/etc/init.d/elasticsearch start (code=exited, status=1/FAILURE)

    Feb 24 08:36:43 SELKS systemd[1]: Starting LSB: Starts elasticsearch...
    Feb 24 08:36:43 SELKS elasticsearch[3606]: The elasticsearch startup script does not exists or it is not executable, tried:
    Feb 24 08:36:43 SELKS systemd[1]: elasticsearch.service: Control process exited, code=exited, status=1/FAILURE
    Feb 24 08:36:43 SELKS systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
    Feb 24 08:36:43 SELKS systemd[1]: Failed to start LSB: Starts elasticsearch.
    selks-user@SELKS:~$ sudo systemctl start elasticsearch
    Job for elasticsearch.service failed because the control process exited with error code.
    See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
    selks-user@SELKS:~$ sudo service elasticsearch start
    Job for elasticsearch.service failed because the control process exited with error code.
    See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
    selks-user@SELKS:~$ sudo journalctl -xe
    Feb 24 08:40:04 SELKS elasticsearch[3693]: The elasticsearch startup script does not exists or it is not executable, tried:
    Feb 24 08:40:04 SELKS systemd[1]: elasticsearch.service: Control process exited, code=exited, status=1/FAILURE
    -- Subject: Unit process exited
    -- Defined-By: systemd
    -- Support: https://www.debian.org/support
    --
    -- An ExecStart= process belonging to unit elasticsearch.service has exited.
    --
    -- The process' exit code is 'exited' and its exit status is 1.
    Feb 24 08:40:04 SELKS systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
    -- Subject: Unit failed
    -- Defined-By: systemd
    -- Support: https://www.debian.org/support
    --
    -- The unit elasticsearch.service has entered the 'failed' state with result 'exit-code'.
    Feb 24 08:40:04 SELKS systemd[1]: Failed to start LSB: Starts elasticsearch.
    -- Subject: A start job for unit elasticsearch.service has failed
    -- Defined-By: systemd
    -- Support: https://www.debian.org/support
    --
    -- A start job for unit elasticsearch.service has finished with a failure.
    --
    -- The job identifier is 1933 and the job result is failed.
    Feb 24 08:40:04 SELKS sudo[3671]: pam_unix(sudo:session): session closed for user root
    Feb 24 08:40:06 SELKS logstash[516]: [2021-02-24T08:40:06,455][WARN ][logstash.outputs.elasticsearch][main] Attempted to res
    Feb 24 08:40:06 SELKS logstash[516]: [2021-02-24T08:40:06,457][WARN ][logstash.outputs.elasticsearch][main] Attempted to res
    Feb 24 08:40:11 SELKS logstash[516]: [2021-02-24T08:40:11,462][WARN ][logstash.outputs.elasticsearch][main] Attempted to res
    Feb 24 08:40:11 SELKS logstash[516]: [2021-02-24T08:40:11,463][WARN ][logstash.outputs.elasticsearch][main] Attempted to res
    Feb 24 08:40:16 SELKS logstash[516]: [2021-02-24T08:40:16,469][WARN ][logstash.outputs.elasticsearch][main] Attempted to res
    Feb 24 08:40:16 SELKS logstash[516]: [2021-02-24T08:40:16,470][WARN ][logstash.outputs.elasticsearch][main] Attempted to res
    Feb 24 08:40:19 SELKS sudo[3699]: selks-user : TTY=pts/0 ; PWD=/home/selks-user ; USER=root ; COMMAND=/bin/journalctl -xe
    Feb 24 08:40:19 SELKS sudo[3699]: pam_unix(sudo:session): session opened for user root by selks-user(uid=0)
    selks-user@SELKS:~$

I took this up with the folks at SELKS initially, and the response was, "It appears that Elasticsearch is not up?" Kibana is not starting, either, but I see a thread I can try for that issue. I'd rather go through these one at a time.

warkolm · February 25, 2021, 2:43am

Welcome to our community!

What do your Elasticsearch logs show?

PSandusky · February 25, 2021, 3:35pm

Thank you!

Here's what I've got in the log this morning...

selks-user@SELKS:~$ sudo tail -100 /var/log/elasticsearch/elasticsearch.log
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [lang-painless]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [mapper-extras]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [mapper-version]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [parent-join]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [percolator]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [rank-eval]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [reindex]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [repositories-metering-api]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [repository-url]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [search-business-rules]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [searchable-snapshots]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [spatial]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [systemd]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [transform]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [transport-netty4]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [unsigned-long]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [vectors]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [wildcard]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-aggregate-metric]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-analytics]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-async]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-async-search]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-autoscaling]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ccr]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-core]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-data-streams]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-deprecation]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-enrich]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-eql]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-fleet]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-graph]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-identity-provider]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ilm]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ingest]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-logstash]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ml]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-monitoring]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-ql]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-rollup]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-runtime-fields]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-security]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-sql]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-stack]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-voting-only-node]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] loaded module [x-pack-watcher]
[2021-02-24T16:38:17,525][INFO ][o.e.p.PluginsService     ] [SELKS] no plugins loaded
[2021-02-24T16:38:17,795][INFO ][o.e.e.NodeEnvironment    ] [SELKS] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [435.4gb], net total_space [483.2gb], types [ext4]
[2021-02-24T16:38:17,796][INFO ][o.e.e.NodeEnvironment    ] [SELKS] heap size [3.8gb], compressed ordinary object pointers [true]
[2021-02-24T16:38:18,026][INFO ][o.e.n.Node               ] [SELKS] node name [SELKS], node ID [voxRMqkTTqGjtgSKpUG0FA], cluster name [elasticsearch], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2021-02-24T16:38:25,224][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [SELKS] [controller/3424] [Main.cc@117] controller (64 bit): Version 7.11.1 (Build b7aec245e3d54f) Copyright (c) 2021 Elasticsearch BV
[2021-02-24T16:38:26,122][INFO ][o.e.x.s.a.s.FileRolesStore] [SELKS] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2021-02-24T16:38:27,651][INFO ][o.e.t.NettyAllocator     ] [SELKS] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2021-02-24T16:38:27,746][INFO ][o.e.d.DiscoveryModule    ] [SELKS] using discovery type [zen] and seed hosts providers [settings]
[2021-02-24T16:38:28,543][INFO ][o.e.g.DanglingIndicesState] [SELKS] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-02-24T16:38:29,355][INFO ][o.e.n.Node               ] [SELKS] initialized
[2021-02-24T16:38:29,356][INFO ][o.e.n.Node               ] [SELKS] starting ...
[2021-02-24T16:38:29,423][INFO ][o.e.x.s.c.PersistentCache] [SELKS] persistent cache index loaded
[2021-02-24T16:38:29,554][INFO ][o.e.t.TransportService   ] [SELKS] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2021-02-24T16:38:29,794][ERROR][o.e.b.Bootstrap          ] [SELKS] Exception
java.lang.IllegalStateException: The index [[logstash-http-2021.02.22/GCm01ePnSx6Ypg84hnKdxQ]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1.
        at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkSupportedVersion(MetadataIndexUpgradeService.java:108) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:82) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:236) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:219) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:150) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.node.Node.start(Node.java:827) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:310) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:396) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.11.1.jar:7.11.1]
[2021-02-24T16:38:29,799][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [SELKS] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: The index [[logstash-http-2021.02.22/GCm01ePnSx6Ypg84hnKdxQ]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1.
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.11.1.jar:7.11.1]
        at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.11.1.jar:7.11.1]
Caused by: java.lang.IllegalStateException: The index [[logstash-http-2021.02.22/GCm01ePnSx6Ypg84hnKdxQ]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1.
        at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.checkSupportedVersion(MetadataIndexUpgradeService.java:108) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:82) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:236) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:219) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:150) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.node.Node.start(Node.java:827) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:310) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:396) ~[elasticsearch-7.11.1.jar:7.11.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.1.jar:7.11.1]
        ... 6 more
[2021-02-24T16:38:29,805][INFO ][o.e.n.Node               ] [SELKS] stopping ...
[2021-02-24T16:38:29,838][INFO ][o.e.n.Node               ] [SELKS] stopped
[2021-02-24T16:38:29,839][INFO ][o.e.n.Node               ] [SELKS] closing ...
[2021-02-24T16:38:29,872][INFO ][o.e.n.Node               ] [SELKS] closed
[2021-02-24T16:38:29,876][INFO ][o.e.x.m.p.NativeController] [SELKS] Native controller process has stopped - no new native processes can be started
selks-user@SELKS:~$

dadoonet · February 25, 2021, 3:57pm

I believe this explains things:

The index [[logstash-http-2021.02.22/GCm01ePnSx6Ypg84hnKdxQ]] was created with version [5.6.16] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.11.1.

Did you upgrade from 5.x to 6.x and then to 7.x without running the migration assistant?

PSandusky · February 25, 2021, 4:29pm

I didn't have anything in the SELKS documentation that describes initiating that process. I know it runs what their scripts refer to as migrations, but clearly this one wasn't in that list.

I'm hunting around for elasticsearch-migrate, but that binary doesn't appear to be in this install. How would I go about getting it into place?

dadoonet · February 25, 2021, 5:00pm

What is SELKS?

It's part of the default 6.8 distribution in Kibana. See Upgrade Assistant | Kibana Guide [6.8] | Elastic

PSandusky · February 25, 2021, 8:19pm

SELKS is a custom Debian distribution that combines an ELK stack with Suricata and Moloch in their own custom web interface. They place great weight on their install and upgrade scripts making everything live out of the box, but there are a couple gaps... to say the least.

Until everything is flying here (?!!), my access to this machine is by SSH. Does the migration system have a headless equivalent?

dadoonet · February 26, 2021, 1:22am

Do you mean that there's no Kibana you can access?

system · March 26, 2021, 1:23am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.