Clean install of ES 7.13.2 on Ubuntu 20.04 hangs

Elastic Stack Elasticsearch
1

I've been trying to test a clean config of ES stack on an Ubuntu VM following Install Elasticsearch with Debian Package | Elasticsearch Guide [7.13] | Elastic

The system was a clean install to a 4 core 4GB VM on 20.04.2 and kernel 5.4.0-74-generic

only other installs to the system were:
default-jre
openssh-server
nginx
kibana

$ java -version
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing)

I've tried both through sudo apt-get install elasticsearch and sudo dpkg -i elasticsearch-7.13.2-amd64.deb after doing a sha512 hash check. Both with and without default-jre installed (using the package JDK with elasticsearch).

Running with sudo dpkg -D777 -i elasticsearch-7.13.2-amd64.deb, the install stops at:

D000200: conffderef in='/etc/default/elasticsearch' current working='/etc/default/elasticsearch'
D000200: conffderef nonexistent
D000020: deferred_configure '/etc/default/elasticsearch' (= '/etc/default/elasticsearch') useredited=-1 distedited=-1 what=24
D000200: conffderef in='/etc/init.d/elasticsearch' current working='/etc/init.d/elasticsearch'
D000200: conffderef nonexistent
D000020: deferred_configure '/etc/init.d/elasticsearch' (= '/etc/init.d/elasticsearch') useredited=-1 distedited=-1 what=24
D000002: fork/exec /var/lib/dpkg/info/elasticsearch.postinst ( configure  )

and htop has about 14 child processes spawned with the command:
/usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Xms4m -Xmx64m -XX:+UseSerialGC -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=deb -cp /usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/tools/keystore-cli/* org.elasticsearch.common.settings.KeyStoreCli create

one of which spins at 100% CPU until killed.

The crash log from the install:

/usr/share/elasticsearch$ cat hs_err_pid5740.log
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fb5e0b4ddfd, pid=5740, tid=5786
#
# JRE version: OpenJDK Runtime Environment AdoptOpenJDK (16.0+36) (build 16+36)
# Java VM: OpenJDK 64-Bit Server VM AdoptOpenJDK (16+36, mixed mode, sharing, tiered, compressed oops, serial gc, linux-amd64)
# Problematic frame:
# J 77 c1 java.lang.Math.max(II)I java.base@16 (11 bytes) @ 0x00007fb5e0b4ddfd [0x00007fb5e0b4de20+0xffffffffffffffdd]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /usr/share/elasticsearch/core.5740)
#
# If you would like to submit a bug report, please visit:
#   https://github.com/AdoptOpenJDK/openjdk-support/issues
#

---------------  S U M M A R Y ------------

Command Line: -Xshare:auto -Xms4m -Xmx64m -XX:+UseSerialGC -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=deb org.elasticsearch.common.settings.KeyStoreCli create

Host: QEMU Virtual CPU version 2.5+, 4 cores, 3G, Ubuntu 20.04.2 LTS
Time: Wed Jun 23 09:53:15 2021 EDT elapsed time: 2.643542 seconds (0d 0h 0m 2s)

---------------  T H R E A D  ---------------

Current thread (0x00007fb5f8022ee0):  JavaThread "main" [_thread_in_Java, id=5786, stack(0x00007fb5fe7c6000,0x00007fb5fe8c7000)]

Stack: [0x00007fb5fe7c6000,0x00007fb5fe8c7000],  sp=0x00007fb5fe8c44b8,  free space=1017k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 77 c1 java.lang.Math.max(II)I java.base@16 (11 bytes) @ 0x00007fb5e0b4ddfd [0x00007fb5e0b4de20+0xffffffffffffffdd]

[error occurred during error reporting (printing native stack), id 0xb, SIGSEGV (0xb) at pc=0x00007fb5ff10f503]


siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x000000086466b099

Register to memory mapping:
2

Just to be clear, it's installed but then fails on startup?
What do the Elasticsearch logs show?

3

It fails during the install, it either dumps a Java error or hangs with a Java process that needs to be killed as part of the post-installation scripts. I’ve tried 7.13.2, 7.13.1, 7.13.0, and 7.6.2 deb packages and all do the same thing so far.

There is no elasticsearch log as it never installs, hence I’m not able to start it.

For example:

ubuntu@es-master1:~$ sudo dpkg -i elasticsearch-7.6.2-amd64.deb
[sudo] password for ubuntu:
Selecting previously unselected package elasticsearch.
(Reading database ... 173281 files and directories currently installed.)
Preparing to unpack elasticsearch-7.6.2-amd64.deb ...
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Unpacking elasticsearch (7.6.2) ...
Setting up elasticsearch (7.6.2) ...
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f7d4cd3f780, pid=6660, tid=6686
#
# JRE version: OpenJDK Runtime Environment (13.0.2+8) (build 13.0.2+8)
# Java VM: OpenJDK 64-Bit Server VM (13.0.2+8, mixed mode, sharing, tiered, compressed oops, serial gc, linux-amd64)
# Problematic frame:
# J 76 c1 java.util.zip.ZipUtils.CENEXT([BI)I java.base@13.0.2 (9 bytes) @ 0x00007f7d4cd3f780 [0x00007f7d4cd3f740+0x0000000000000040]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /usr/share/elasticsearch/core.6660)
#
# An error report file with more information is saved as:
# /usr/share/elasticsearch/hs_err_pid6660.log
Compiled method (c1)    2275   76       3       java.util.zip.ZipUtils::CENEXT (9 bytes)
 total in heap  [0x00007f7d4cd3f590,0x00007f7d4cd3fa10] = 1152
 relocation     [0x00007f7d4cd3f6f0,0x00007f7d4cd3f728] = 56
 main code      [0x00007f7d4cd3f740,0x00007f7d4cd3f8a0] = 352
 stub code      [0x00007f7d4cd3f8a0,0x00007f7d4cd3f930] = 144
 metadata       [0x00007f7d4cd3f930,0x00007f7d4cd3f940] = 16
 scopes data    [0x00007f7d4cd3f940,0x00007f7d4cd3f970] = 48
 scopes pcs     [0x00007f7d4cd3f970,0x00007f7d4cd3f9f0] = 128
 dependencies   [0x00007f7d4cd3f9f0,0x00007f7d4cd3f9f8] = 8
 nul chk table  [0x00007f7d4cd3f9f8,0x00007f7d4cd3fa10] = 24
#
# If you would like to submit a bug report, please visit:
#   https://github.com/AdoptOpenJDK/openjdk-support/issues
#
/usr/share/elasticsearch/bin/elasticsearch-keystore: line 5:  6660 Aborted                 (core dumped) ES_MAIN_CLASS=org.elasticsearch.common.settings.KeyStoreCli "`dirname "$0"`"/elasticsearch-cli "$@"
dpkg: error processing package elasticsearch (--install):
 installed elasticsearch package post-installation script subprocess returned error exit status 134
Processing triggers for systemd (245.4-4ubuntu3.7) ...
Errors were encountered while processing:
 elasticsearch
ubuntu@es-master1:~$

The contents of that file are:

ubuntu@es-master1:/usr/share/elasticsearch/bin$ cat elasticsearch-keystore
#!/bin/bash

ES_MAIN_CLASS=org.elasticsearch.common.settings.KeyStoreCli \
  "`dirname "$0"`"/elasticsearch-cli \
  "$@"
ubuntu@es-master1:/usr/share/elasticsearch/bin$

When this fails, it leaves java processes running:

image
image1544×815 92.6 KB

4

the problem sounds very similar to Working with elasticsearch-keystore

and possibly Docs say elasticsearch-keystore must be run as the elasticsearch user but that fails because of permissions · Issue #66133 · elastic/elasticsearch · GitHub

it's getting to /var/lib/dpkg/info/elasticsearch.postinst and hanging at the elasticsearch-keystore create in this seciton of that file:

# the equivalent code for rpm is in posttrans
if [ "$PACKAGE" = "deb" ]; then
    if [ ! -f "${ES_PATH_CONF}"/elasticsearch.keystore ]; then
        /usr/share/elasticsearch/bin/elasticsearch-keystore create
        chown root:elasticsearch "${ES_PATH_CONF}"/elasticsearch.keystore
        chmod 660 "${ES_PATH_CONF}"/elasticsearch.keystore
        md5sum "${ES_PATH_CONF}"/elasticsearch.keystore > "${ES_PATH_CONF}"/.elasticsearch.keystore.initial_md5sum
    else
        if /usr/share/elasticsearch/bin/elasticsearch-keystore has-passwd --silent ; then
          echo "### Warning: unable to upgrade encrypted keystore" 1>&2
          echo " Please run elasticsearch-keystore upgrade and enter password" 1>&2
        else
          /usr/share/elasticsearch/bin/elasticsearch-keystore upgrade
        fi
    fi
fi

exit 0
# Built for packages-7.13.2 (deb)

ubuntu@es-master1:~$
5

so, this seems to be a concurrency check problem, I've overriden ES_JAVA_HOME and finally get to this:

D000002: fork/exec /var/lib/dpkg/info/elasticsearch.postinst ( configure  )
Future versions of Elasticsearch will require Java 11; your Java version from [/usr/local/java/jre1.8.0_291] does not meet this requirement. Consider switching to a distribution of Elasticsearch with a bundled JDK. If you are already using a distribution with a bundled JDK, ensure the JAVA_HOME environment variable is not set.
Exception in thread "main" java.nio.file.FileAlreadyExistsException: /etc/elasticsearch/elasticsearch.keystore.tmp
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:88)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
        at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
        at java.nio.file.Files.newOutputStream(Files.java:216)
        at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:410)
        at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:406)
        at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:254)
        at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:484)
        at org.elasticsearch.common.settings.CreateKeyStoreCommand.execute(CreateKeyStoreCommand.java:46)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
        at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:80)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
        at org.elasticsearch.cli.Command.main(Command.java:79)
        at org.elasticsearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:32)
dpkg: error processing package elasticsearch (--install):
 installed elasticsearch package post-installation script subprocess returned error exit status 1

I'm completely at a loss at what to do to get this working.

6

update

so the install of ES on the host machine at 20.04.2 works fine, just the install in the qemu-kvm guest is hanging with this java error on creating the keystore.

thoughts?

7

UPDATE 2:

after digging through various libvirt/kvm configs, I figured out the problem. For some reason libvirt defaulted to generate a domain cpu config of:

  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>EPYC</model>
    <vendor>AMD</vendor>
    <feature policy='require' name='acpi'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='erms'/>
    <feature policy='require' name='mpx'/>
    <feature policy='require' name='pcommit'/>
    <feature policy='require' name='clwb'/>
    <feature policy='require' name='pku'/>
    <feature policy='require' name='la57'/>
    <feature policy='require' name='3dnowext'/>
    <feature policy='require' name='3dnow'/>
    <feature policy='require' name='npt'/>
    <feature policy='disable' name='vme'/>
    ...

When the host cpu was an Intel Xeon with VT-x enabled. After changing the config with virsh edit <domain> and then SHUTTING DOWN the domain, i.e. NOT a restart, it installed properly. Thanks to virsh edit vs virsh dumpxml - Red Hat Customer Portal for this little nuance. So now a dumpxml of the domain shows:

  <cpu mode='host-passthrough' check='none'/>
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.