Cannot start Elasticsearch service when installing Elasticsearch with Debian package on Linux Mint localhost

Elastic Stack Elasticsearch
1

I installed Eleasticsearch following the tutorial: Install Elasticsearch with Debian Package | Elasticsearch Guide [8.3] | Elastic

Everything installs perfectly:

Unpacking elasticsearch (8.3.2) ...
Setting up elasticsearch (8.3.2) ...
--------------------------- Security autoconfiguration information ------------------------------

Skipping auto-configuration because security features appear to be already configured.

-------------------------------------------------------------------------------------------------
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service
Created elasticsearch keystore in /etc/elasticsearch/elasticsearch.keystor

Step 1:

marius@marius:~$ sudo systemctl daemon-reload
marius@marius:~$ sudo systemctl enable elasticsearch.service
Created symlink /etc/systemd/system/multi-user.target.wants/elasticsearch.service → /lib/systemd/system/elasticsearch.service.
marius@marius:~$ sudo systemctl start elasticsearch.service
Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

Step 2:
marius@marius:~$ systemctl status elasticsearch.service
Result:

● elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2022-07-14 07:55:08 EEST; 16s ago
       Docs: https://www.elastic.co
    Process: 6862 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
   Main PID: 6862 (code=exited, status=1/FAILURE)

Jul 14 07:55:08 marius systemd-entrypoint[6862]:         at org.elasticsearch.xpack.security.cli.AutoConfigureNode.execute(AutoConfigureNode.java:166)
Jul 14 07:55:08 marius systemd-entrypoint[6862]:         at org.elasticsearch.server.cli.ServerCli.autoConfigureSecurity(ServerCli.java:149)
Jul 14 07:55:08 marius systemd-entrypoint[6862]:         at org.elasticsearch.server.cli.ServerCli.execute(ServerCli.java:87)
Jul 14 07:55:08 marius systemd-entrypoint[6862]:         at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54)
Jul 14 07:55:08 marius systemd-entrypoint[6862]:         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
Jul 14 07:55:08 marius systemd-entrypoint[6862]:         at org.elasticsearch.cli.Command.main(Command.java:50)
Jul 14 07:55:08 marius systemd-entrypoint[6862]:         at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64)
Jul 14 07:55:08 marius systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Jul 14 07:55:08 marius systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Jul 14 07:55:08 marius systemd[1]: Failed to start Elasticsearch.

Step 3:
marius@marius:~$ journalctl -u elasticsearch.service
Result:

-- Logs begin at Thu 2022-07-14 07:42:08 EEST, end at Thu 2022-07-14 07:58:19 EEST. --
Jul 14 07:55:07 marius systemd[1]: Starting Elasticsearch...
Jul 14 07:55:08 marius systemd-entrypoint[6862]: Exception in thread "main" java.nio.file.AccessDeniedException: /var/lib/elasticsearch
Jul 14 07:55:08 marius systemd-entrypoint[6862]:         at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
Jul 14 07:55:08 marius systemd-entrypoint[6862]:         at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
Jul 14 07:55:08 marius systemd-entrypoint[6862]:         at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
Jul 14 07:55:08 marius systemd-entrypoint[6862]:         at java.base/sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:440)
Jul 14 07:55:08 marius systemd-entrypoint[6862]:         at java.base/java.nio.file.Files.newDirectoryStream(Files.java:482)
Jul 14 07:55:08 marius systemd-entrypoint[6862]:         at java.base/java.nio.file.Files.list(Files.java:3793)
Jul 14 07:55:08 marius systemd-entrypoint[6862]:         at org.elasticsearch.xpack.security.cli.AutoConfigureNode.isDirEmpty(AutoConfigureNode.java:1144)
Jul 14 07:55:08 marius systemd-entrypoint[6862]:         at org.elasticsearch.xpack.security.cli.AutoConfigureNode.execute(AutoConfigureNode.java:166)
Jul 14 07:55:08 marius systemd-entrypoint[6862]:         at org.elasticsearch.server.cli.ServerCli.autoConfigureSecurity(ServerCli.java:149)
Jul 14 07:55:08 marius systemd-entrypoint[6862]:         at org.elasticsearch.server.cli.ServerCli.execute(ServerCli.java:87)
Jul 14 07:55:08 marius systemd-entrypoint[6862]:         at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54)
Jul 14 07:55:08 marius systemd-entrypoint[6862]:         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
Jul 14 07:55:08 marius systemd-entrypoint[6862]:         at org.elasticsearch.cli.Command.main(Command.java:50)
Jul 14 07:55:08 marius systemd-entrypoint[6862]:         at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64)
Jul 14 07:55:08 marius systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Jul 14 07:55:08 marius systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Jul 14 07:55:08 marius systemd[1]: Failed to start Elasticsearch.

Step 4:
I've changed permisions in different ways to first to elesticsearch after to user "marius", so not root but the elasticsearch service do not start.

marius@marius:/etc$ sudo chmod +x elasticsearch
marius@marius:/etc$ sudo ls -l elasticsearch
total 48
-rw-rw---- 1 root elasticsearch   199 Jul 14 07:53 elasticsearch.keystore
-rw-rw---- 1 root elasticsearch  1042 Jul  6 18:22 elasticsearch-plugins.example.yml
-rw-rw---- 1 root elasticsearch  2919 Jul  6 18:22 elasticsearch.yml
-rw-rw---- 1 root elasticsearch  2617 Jul  6 18:22 jvm.options
drwxr-s--- 2 root elasticsearch  4096 Jul  6 18:22 jvm.options.d
-rw-rw---- 1 root elasticsearch 16644 Jul  6 18:22 log4j2.properties
-rw-rw---- 1 root elasticsearch   473 Jul  6 18:22 role_mapping.yml
-rw-rw---- 1 root elasticsearch   197 Jul  6 18:22 roles.yml
-rw-rw---- 1 root elasticsearch     0 Jul  6 18:22 users
-rw-rw---- 1 root elasticsearch     0 Jul  6 18:22 users_roles
marius@marius:/etc$ sudo chown -R elasticsearch:elasticsearch elasticsearch
marius@marius:/etc$ sudo ls -l elasticsearch
total 48
-rw-rw---- 1 elasticsearch elasticsearch   199 Jul 14 07:53 elasticsearch.keystore
-rw-rw---- 1 elasticsearch elasticsearch  1042 Jul  6 18:22 elasticsearch-plugins.example.yml
-rw-rw---- 1 elasticsearch elasticsearch  2919 Jul  6 18:22 elasticsearch.yml
-rw-rw---- 1 elasticsearch elasticsearch  2617 Jul  6 18:22 jvm.options
drwxr-s--- 2 elasticsearch elasticsearch  4096 Jul  6 18:22 jvm.options.d
-rw-rw---- 1 elasticsearch elasticsearch 16644 Jul  6 18:22 log4j2.properties
-rw-rw---- 1 elasticsearch elasticsearch   473 Jul  6 18:22 role_mapping.yml
-rw-rw---- 1 elasticsearch elasticsearch   197 Jul  6 18:22 roles.yml
-rw-rw---- 1 elasticsearch elasticsearch     0 Jul  6 18:22 users
-rw-rw---- 1 elasticsearch elasticsearch     0 Jul  6 18:22 users_roles
marius@marius:/etc$ sudo chown -R marius:marius elasticsearch
marius@marius:/etc$ sudo ls -l elasticsearch
total 48
-rw-rw---- 1 marius marius   199 Jul 14 07:53 elasticsearch.keystore
-rw-rw---- 1 marius marius  1042 Jul  6 18:22 elasticsearch-plugins.example.yml
-rw-rw---- 1 marius marius  2919 Jul  6 18:22 elasticsearch.yml
-rw-rw---- 1 marius marius  2617 Jul  6 18:22 jvm.options
drwxr-s--- 2 marius marius  4096 Jul  6 18:22 jvm.options.d
-rw-rw---- 1 marius marius 16644 Jul  6 18:22 log4j2.properties
-rw-rw---- 1 marius marius   473 Jul  6 18:22 role_mapping.yml
-rw-rw---- 1 marius marius   197 Jul  6 18:22 roles.yml
-rw-rw---- 1 marius marius     0 Jul  6 18:22 users
-rw-rw---- 1 marius marius     0 Jul  6 18:22 users_roles

Step 5:
I try to see full logs but no logs. Also I discovered that almost everything related to Elasticsearch is Permission denied for user:

marius@marius:~$ cd /var/log/elasticsearch
bash: cd: /var/log/elasticsearch: Permission denied
marius@marius:~$ sudo bash
[sudo] password for marius:           
root@marius:/home/marius# cd /var/log/elasticsearch
root@marius:/var/log/elasticsearch# ls
root@marius:/var/log/elasticsearch# ls -la
total 8
drwxr-s---  2 elasticsearch elasticsearch 4096 Jul  6 18:22 .
drwxrwxr-x 11 root          syslog        4096 Jul 14 07:53 ..

Elasticsearch works fine on Linux Mint 20.3 if installed from the archive option, I tested this. But I need to install via the deb package to be able to automatically start the service on boot, symlink is not possible if installed via the archive option for autostart on boot.

Question: Does anyone have any clue how to start Eleasticsearch if it is installed via .deb package?

2

Couple initial questions

When you installed the.deb package did you use sudo and all the defaults?

I am curious why you got this message

Skipping auto-configuration because security features appear to be already configured.

That appears that you are reinstalling over and existing installation? Is that what you are doing?

If you run

sudo dpkg -i elasticsearch-8.3.2

On a fresh install it should setup all the default security and all the proper permissions etc.

Is it possible there was a previous install or some of your earlier attempts to install left some previous files etc?

3

  1. Yes I use sudo when installing (In Linux Mint it is impossible to install anything without sudo). Even Elasticsearch tutorial is with sudo ( sudo apt-get update && sudo apt-get install elasticsearch).

  2. I've try before several times to install Elasticsearch but always I get different errors. But fefore each fresh install I use:

sudo apt-get --purge autoremove elasticsearch
sudo apt-get remove --purge elasticsearch
sudo rm -rf /etc/elasticsearch
  1. marius@marius:/etc$ sudo dpkg -i elasticsearch-8.3.2

Result:

dpkg: error: cannot access archive 'elasticsearch-8.3.2': No such file or directory
dpkg: error: cannot access archive 'elasticsearch-8.3.2': No such file or directory
4

Skipping auto-configuration because

The message above indicates that there's leftover data still. I know purge is supposed to remove all that but perhaps it's not.

I would check the data directory and make sure that is empty as well. I suspect it is not.
You can see where the default directories for the deb install is here

In particular, you should check the data directory.

data The location of the data files of each index / shard allocated on the node. /var/lib/elasticsearch

To install .deb manually see here

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.3.2-amd64.deb
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.3.2-amd64.deb.sha512
shasum -a 512 -c elasticsearch-8.3.2-amd64.deb.sha512 
sudo dpkg -i elasticsearch-8.3.2-amd64.deb
5

@stephenb, your suspicions have come true: the /var/lib/elasticsearch has not been purged. After I delete Elasticsearch from this location as well, it reinstalled correctly.

Kibana - the same situation
By the way: the same problem is replicated when reinstalling Kibana if it was previously installed, only here the error message takes you further into the fog; that's because Kibana installs and starts the Kibana service, except that the service status shows you that the service is running but you get an error message when connecting to Elasticsearch, a message that does not exist on Google :slight_smile:
After unistall Kibana and deleting var/lib/kibana the Kibana install correctly.

I finally managed to activate and run Elasticsearch 8.3.2 and Kibana in Linux Mint local host via .deb package.

BUG or misconfiguration?
When you install/reinstall Elasticsearch 8.3.2 and Kibana from scratch via .deb package you face situations that are either BUGs or are a misconfiguration.
Argumentation:
1.Installing Elasticsearch via the .deb package is done using "sudo", however everywhere it is stated that Elasticsearch should not be started as root.

  1. After installing Elasticsearch with root privileges you encounter the following problem:
    Starting Elasticsearch via systemctl does not generate the token. And generating the token via "./elasticsearch-create-enrollment-token -s kibana" will give you "Permission denied" error if you are not root. And Linux Mint (I think Ubuntu too) doesn't have the root user pre-installed, it has the user that is part of the root group. And you cannot use "sudo cd/etc/eleasticsearch" and "cd /etc/elasticsearch is "Permission denied". The only solution to run scripts with root is the "sudo bash" command - to become root in terminal. That way you can access the above folders that the user part of root group previously got on access "Permission denied". And changing permissions on these folders is not recommended, it just won't work correctly (tested).

  2. Activating Kibana via systemctl will not provide you with the e.g "http://localhost:5601/?code=239634" link in the terminal even if you have previously generated the Elasticsearch token. You need to run the command "sudo /bin/systemctl status kibana.service" to get the link in terminal and start using Elasticsearch in the browser (which is not stated anywhere in the documentation).

1 Like
6

Installing elasticsearch .deb package with sudo does not result in elasticsearch running as root, elasticsearch will run as elastic user when started with systemctl. During the package installation the correct elasticsearch users and groups are created. (it is actually runs as user elastic)

This is an install using sudo dpkg -i elasticsearch-8.3.2-amd64.deb

See Below, elasticsearch is running as user elasticsearch

sbrown@stephenb-es-8-test:~$ sudo systemctl start elasticsearch
sbrown@stephenb-es-8-test:~$ sudo systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-07-14 19:04:43 UTC; 27s ago
       Docs: https://www.elastic.co
   Main PID: 197217 (java)
      Tasks: 72 (limit: 4696)
     Memory: 2.4G
     CGroup: /system.slice/elasticsearch.service
             ├─197217 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cach>
             └─197491 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Jul 14 19:04:09 stephenb-es-8-test systemd[1]: Starting Elasticsearch...
Jul 14 19:04:43 stephenb-es-8-test systemd[1]: Started Elasticsearch.

### Note elasticsearch running as user elastic
sbrown@stephenb-es-8-test:~$ ps -ef 
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 Jun27 ?        00:00:56 /sbin/init
....
elastic+  197217       1 69 19:04 ?        00:01:20 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.tt
elastic+  197491  197217  0 19:04 ?        00:00:00 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

root@stephenb-es-8-test:/etc/elasticsearch# cat /etc/passwd | grep elastic
elastic:x:1073:1074::/home/elastic:/bin/bash
elasticsearch:x:113:120::/nonexistent:/bin/false

The kibana enrollment tokens and paswords etc are generated during the package installation not when started with systemctl.

See Here

During installation (when running sudo dpkg -i elasticsearch-8.3.2-amd64.deb ) the following will show up on the terminal, you need to copy / save this, note that it contains the kibana enrollment token etc.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ Elasticsearch security features have been automatically configured!
✅ Authentication is enabled and cluster connections are encrypted.

ℹ️  Password for the elastic user (reset with `bin/elasticsearch-reset-password -u elastic`):
  sdfgsdfgsdfgsdfggsdfg

ℹ️  HTTP CA certificate SHA-256 fingerprint:
  sdsdfgsdfgsdfgsdfg5c6d4ccbfa7a0081a2cfc4c07939b16d9cd937de

ℹ️  Configure Kibana to use this cluster:
• Run Kibana and click the configuration link in the terminal when Kibana starts.
• Copy the following enrollment token and paste it into Kibana in your browser (valid for the next 30 minutes):
  sdfgsdfgsdfgdsfgMTA3OjkyMDAiXSwiZmdyIjoiMTZmNmZiYmNkMTUxY2ZjOGZjZWMyNWU5NWM2ZDRjY2JmYTdhMDA4MWEyY2ZjNGMwNzkzOWIxNmQ5Y2Q5MzdkZSIsImtleSI6IjNPa1ExSUVCSVhadnZOUFlTZ2Z5OlVoR0R5WlUzUnhpYlhyX0otSlQ2MlEifQ==

ℹ️  Configure other nodes to join this cluster:
• On this node:
  ⁃ Create an enrollment token with `bin/elasticsearch-create-enrollment-token -s node`.
  ⁃ Uncomment the transport.host setting at the end of config/elasticsearch.yml.
  ⁃ Restart Elasticsearch.
• On other nodes:
  ⁃ Start Elasticsearch with `bin/elasticsearch --enrollment-token <token>`, using the enrollment token that you generated.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Yup that should probably be
sudo ./bin/elasticsearch-create-enrollment-token -s kibana

Hmm fair enough, that may be the case I will need to look / try again... Pretty sure that link is written into the journal control, I would need to a reinstall from scratch to check the behavior, but yes that may be missing from the docs...

Log information can be accessed via journalctl -u kibana.service .

Glad you got it working

7

@marius03

Stumbled across this today, probably need to be linked into the docs for the .deb / .rpm install

Description

Use this command to retrieve a verification code for Kibana. You enter this code in Kibana when manually configuring a secure connection with an Elasticsearch cluster. This tool is useful if you don’t have access to the Kibana terminal output, such as on a hosted environment. You can connect to a machine where Kibana is running (such as using SSH) and retrieve a verification code that you enter in Kibana.

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.