# Clean install of ES 7.13.2 on Ubuntu 20.04 hangs

**URL:** https://discuss.elastic.co/t/clean-install-of-es-7-13-2-on-ubuntu-20-04-hangs/276808
**Category:** Elasticsearch
**Created:** [June 23, 2021, 2:22pm UTC](https://discuss.elastic.co/t/clean-install-of-es-7-13-2-on-ubuntu-20-04-hangs/276808 "2021-06-23T14:22:57Z")
**Posts on this page:** 8
**Page:** 1

<div class="post-metadata">

### Author: ![Roger\_Clark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roger_clark/32/77615_2.png) [@Roger\_Clark](https://discuss.elastic.co/u/Roger_Clark)
#### Post date: [June 23, 2021, 2:22pm UTC](https://discuss.elastic.co/t/clean-install-of-es-7-13-2-on-ubuntu-20-04-hangs/276808/1 "2021-06-23T14:22:57Z")

</div>

I've been trying to test a clean config of ES stack on an Ubuntu VM following [Install Elasticsearch with Debian Package | Elasticsearch Guide [7.13] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html#deb-repo)

The system was a clean install to a 4 core 4GB VM on 20.04.2 and kernel 5.4.0-74-generic

only other installs to the system were:  
default-jre  
openssh-server  
nginx  
kibana

```auto
$ java -version
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing)

```

I've tried both through `sudo apt-get install elasticsearch` and `sudo dpkg -i elasticsearch-7.13.2-amd64.deb` after doing a sha512 hash check. Both with and without default-jre installed (using the package JDK with elasticsearch).

Running with `sudo dpkg -D777 -i elasticsearch-7.13.2-amd64.deb`, the install stops at:

```auto
D000200: conffderef in='/etc/default/elasticsearch' current working='/etc/default/elasticsearch'
D000200: conffderef nonexistent
D000020: deferred_configure '/etc/default/elasticsearch' (= '/etc/default/elasticsearch') useredited=-1 distedited=-1 what=24
D000200: conffderef in='/etc/init.d/elasticsearch' current working='/etc/init.d/elasticsearch'
D000200: conffderef nonexistent
D000020: deferred_configure '/etc/init.d/elasticsearch' (= '/etc/init.d/elasticsearch') useredited=-1 distedited=-1 what=24
D000002: fork/exec /var/lib/dpkg/info/elasticsearch.postinst ( configure )

```

and `htop` has about 14 child processes spawned with the command:  
`/usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Xms4m -Xmx64m -XX:+UseSerialGC -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=deb -cp /usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/tools/keystore-cli/* org.elasticsearch.common.settings.KeyStoreCli create`

one of which spins at 100% CPU until killed.

The crash log from the install:

```auto
/usr/share/elasticsearch$ cat hs_err_pid5740.log
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fb5e0b4ddfd, pid=5740, tid=5786
#
# JRE version: OpenJDK Runtime Environment AdoptOpenJDK (16.0+36) (build 16+36)
# Java VM: OpenJDK 64-Bit Server VM AdoptOpenJDK (16+36, mixed mode, sharing, tiered, compressed oops, serial gc, linux-amd64)
# Problematic frame:
# J 77 c1 java.lang.Math.max(II)I java.base@16 (11 bytes) @ 0x00007fb5e0b4ddfd [0x00007fb5e0b4de20+0xffffffffffffffdd]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /usr/share/elasticsearch/core.5740)
#
# If you would like to submit a bug report, please visit:
# https://github.com/AdoptOpenJDK/openjdk-support/issues
#

--------------- S U M M A R Y ------------

Command Line: -Xshare:auto -Xms4m -Xmx64m -XX:+UseSerialGC -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=deb org.elasticsearch.common.settings.KeyStoreCli create

Host: QEMU Virtual CPU version 2.5+, 4 cores, 3G, Ubuntu 20.04.2 LTS
Time: Wed Jun 23 09:53:15 2021 EDT elapsed time: 2.643542 seconds (0d 0h 0m 2s)

--------------- T H R E A D ---------------

Current thread (0x00007fb5f8022ee0): JavaThread "main" [_thread_in_Java, id=5786, stack(0x00007fb5fe7c6000,0x00007fb5fe8c7000)]

Stack: [0x00007fb5fe7c6000,0x00007fb5fe8c7000], sp=0x00007fb5fe8c44b8, free space=1017k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 77 c1 java.lang.Math.max(II)I java.base@16 (11 bytes) @ 0x00007fb5e0b4ddfd [0x00007fb5e0b4de20+0xffffffffffffffdd]

[error occurred during error reporting (printing native stack), id 0xb, SIGSEGV (0xb) at pc=0x00007fb5ff10f503]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x000000086466b099

Register to memory mapping:

```

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [June 23, 2021, 9:29pm UTC](https://discuss.elastic.co/t/clean-install-of-es-7-13-2-on-ubuntu-20-04-hangs/276808/2 "2021-06-23T21:29:27Z")

</div>

Just to be clear, it's installed but then fails on startup?  
What do the Elasticsearch logs show?

---

<div class="post-metadata">

### Author: ![Roger\_Clark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roger_clark/32/77615_2.png) [@Roger\_Clark](https://discuss.elastic.co/u/Roger_Clark)
#### Post date: [June 23, 2021, 11:22pm UTC](https://discuss.elastic.co/t/clean-install-of-es-7-13-2-on-ubuntu-20-04-hangs/276808/3 "2021-06-23T23:22:29Z")

</div>

It fails during the install, it either dumps a Java error or hangs with a Java process that needs to be killed as part of the post-installation scripts. I’ve tried 7.13.2, 7.13.1, 7.13.0, and 7.6.2 deb packages and all do the same thing so far.

There is no elasticsearch log as it never installs, hence I’m not able to start it.

For example:

```auto
ubuntu@es-master1:~$ sudo dpkg -i elasticsearch-7.6.2-amd64.deb
[sudo] password for ubuntu:
Selecting previously unselected package elasticsearch.
(Reading database ... 173281 files and directories currently installed.)
Preparing to unpack elasticsearch-7.6.2-amd64.deb ...
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Unpacking elasticsearch (7.6.2) ...
Setting up elasticsearch (7.6.2) ...
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f7d4cd3f780, pid=6660, tid=6686
#
# JRE version: OpenJDK Runtime Environment (13.0.2+8) (build 13.0.2+8)
# Java VM: OpenJDK 64-Bit Server VM (13.0.2+8, mixed mode, sharing, tiered, compressed oops, serial gc, linux-amd64)
# Problematic frame:
# J 76 c1 java.util.zip.ZipUtils.CENEXT([BI)I java.base@13.0.2 (9 bytes) @ 0x00007f7d4cd3f780 [0x00007f7d4cd3f740+0x0000000000000040]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /usr/share/elasticsearch/core.6660)
#
# An error report file with more information is saved as:
# /usr/share/elasticsearch/hs_err_pid6660.log
Compiled method (c1) 2275 76 3 java.util.zip.ZipUtils::CENEXT (9 bytes)
 total in heap [0x00007f7d4cd3f590,0x00007f7d4cd3fa10] = 1152
 relocation [0x00007f7d4cd3f6f0,0x00007f7d4cd3f728] = 56
 main code [0x00007f7d4cd3f740,0x00007f7d4cd3f8a0] = 352
 stub code [0x00007f7d4cd3f8a0,0x00007f7d4cd3f930] = 144
 metadata [0x00007f7d4cd3f930,0x00007f7d4cd3f940] = 16
 scopes data [0x00007f7d4cd3f940,0x00007f7d4cd3f970] = 48
 scopes pcs [0x00007f7d4cd3f970,0x00007f7d4cd3f9f0] = 128
 dependencies [0x00007f7d4cd3f9f0,0x00007f7d4cd3f9f8] = 8
 nul chk table [0x00007f7d4cd3f9f8,0x00007f7d4cd3fa10] = 24
#
# If you would like to submit a bug report, please visit:
# https://github.com/AdoptOpenJDK/openjdk-support/issues
#
/usr/share/elasticsearch/bin/elasticsearch-keystore: line 5: 6660 Aborted (core dumped) ES_MAIN_CLASS=org.elasticsearch.common.settings.KeyStoreCli "`dirname "$0"`"/elasticsearch-cli "$@"
dpkg: error processing package elasticsearch (--install):
 installed elasticsearch package post-installation script subprocess returned error exit status 134
Processing triggers for systemd (245.4-4ubuntu3.7) ...
Errors were encountered while processing:
 elasticsearch
ubuntu@es-master1:~$

```

The contents of that file are:

```auto
ubuntu@es-master1:/usr/share/elasticsearch/bin$ cat elasticsearch-keystore
#!/bin/bash

ES_MAIN_CLASS=org.elasticsearch.common.settings.KeyStoreCli \
  "`dirname "$0"`"/elasticsearch-cli \
  "$@"
ubuntu@es-master1:/usr/share/elasticsearch/bin$

```

When this fails, it leaves `java` processes running:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/3/0321bf87a71847b04f6b62d721ba66bf9dda9ecd.png)

---

<div class="post-metadata">

### Author: ![Roger\_Clark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roger_clark/32/77615_2.png) [@Roger\_Clark](https://discuss.elastic.co/u/Roger_Clark)
#### Post date: [June 24, 2021, 1:39pm UTC](https://discuss.elastic.co/t/clean-install-of-es-7-13-2-on-ubuntu-20-04-hangs/276808/4 "2021-06-24T13:39:15Z")

</div>

the problem sounds very similar to [Working with elasticsearch-keystore](https://discuss.elastic.co/t/working-with-elasticsearch-keystore/234429)

and possibly [Docs say elasticsearch-keystore must be run as the elasticsearch user but that fails because of permissions · Issue #66133 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/66133)

it's getting to `/var/lib/dpkg/info/elasticsearch.postinst` and hanging at the `elasticsearch-keystore create` in this seciton of that file:

```auto
# the equivalent code for rpm is in posttrans
if ["$PACKAGE" = "deb"]; then
    if [! -f "${ES_PATH_CONF}"/elasticsearch.keystore]; then
        /usr/share/elasticsearch/bin/elasticsearch-keystore create
        chown root:elasticsearch "${ES_PATH_CONF}"/elasticsearch.keystore
        chmod 660 "${ES_PATH_CONF}"/elasticsearch.keystore
        md5sum "${ES_PATH_CONF}"/elasticsearch.keystore > "${ES_PATH_CONF}"/.elasticsearch.keystore.initial_md5sum
    else
        if /usr/share/elasticsearch/bin/elasticsearch-keystore has-passwd --silent ; then
          echo "### Warning: unable to upgrade encrypted keystore" 1>&2
          echo " Please run elasticsearch-keystore upgrade and enter password" 1>&2
        else
          /usr/share/elasticsearch/bin/elasticsearch-keystore upgrade
        fi
    fi
fi

exit 0
# Built for packages-7.13.2 (deb)

ubuntu@es-master1:~$

```

---

<div class="post-metadata">

### Author: ![Roger\_Clark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roger_clark/32/77615_2.png) [@Roger\_Clark](https://discuss.elastic.co/u/Roger_Clark)
#### Post date: [June 24, 2021, 6:39pm UTC](https://discuss.elastic.co/t/clean-install-of-es-7-13-2-on-ubuntu-20-04-hangs/276808/5 "2021-06-24T18:39:34Z")

</div>

so, this seems to be a concurrency check problem, I've overriden ES\_JAVA\_HOME and finally get to this:

```auto
D000002: fork/exec /var/lib/dpkg/info/elasticsearch.postinst ( configure )
Future versions of Elasticsearch will require Java 11; your Java version from [/usr/local/java/jre1.8.0_291] does not meet this requirement. Consider switching to a distribution of Elasticsearch with a bundled JDK. If you are already using a distribution with a bundled JDK, ensure the JAVA_HOME environment variable is not set.
Exception in thread "main" java.nio.file.FileAlreadyExistsException: /etc/elasticsearch/elasticsearch.keystore.tmp
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:88)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
        at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
        at java.nio.file.Files.newOutputStream(Files.java:216)
        at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:410)
        at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:406)
        at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:254)
        at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:484)
        at org.elasticsearch.common.settings.CreateKeyStoreCommand.execute(CreateKeyStoreCommand.java:46)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
        at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:80)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
        at org.elasticsearch.cli.Command.main(Command.java:79)
        at org.elasticsearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:32)
dpkg: error processing package elasticsearch (--install):
 installed elasticsearch package post-installation script subprocess returned error exit status 1

```

I'm completely at a loss at what to do to get this working.

---

<div class="post-metadata">

### Author: ![Roger\_Clark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roger_clark/32/77615_2.png) [@Roger\_Clark](https://discuss.elastic.co/u/Roger_Clark)
#### Post date: [June 25, 2021, 8:51pm UTC](https://discuss.elastic.co/t/clean-install-of-es-7-13-2-on-ubuntu-20-04-hangs/276808/6 "2021-06-25T20:51:42Z")

</div>

update

so the install of ES on the _host_ machine at 20.04.2 works fine, just the install in the qemu-kvm guest is hanging with this java error on creating the keystore.

thoughts?

---

<div class="post-metadata">

### Author: ![Roger\_Clark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roger_clark/32/77615_2.png) [@Roger\_Clark](https://discuss.elastic.co/u/Roger_Clark)
#### Post date: [June 27, 2021, 11:28am UTC](https://discuss.elastic.co/t/clean-install-of-es-7-13-2-on-ubuntu-20-04-hangs/276808/7 "2021-06-27T11:28:02Z")

</div>

UPDATE 2:

after digging through various libvirt/kvm configs, I figured out the problem. For some reason libvirt defaulted to generate a domain cpu config of:

```auto
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>EPYC</model>
    <vendor>AMD</vendor>
    <feature policy='require' name='acpi'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='erms'/>
    <feature policy='require' name='mpx'/>
    <feature policy='require' name='pcommit'/>
    <feature policy='require' name='clwb'/>
    <feature policy='require' name='pku'/>
    <feature policy='require' name='la57'/>
    <feature policy='require' name='3dnowext'/>
    <feature policy='require' name='3dnow'/>
    <feature policy='require' name='npt'/>
    <feature policy='disable' name='vme'/>
    ...

```

When the host cpu was an Intel Xeon with VT-x enabled. After changing the config with `virsh edit <domain>` and then SHUTTING DOWN the domain, i.e. NOT a restart, it installed properly. Thanks to [virsh edit vs virsh dumpxml - Red Hat Customer Portal](https://access.redhat.com/discussions/2720821) for this little nuance. So now a dumpxml of the domain shows:

```auto
  <cpu mode='host-passthrough' check='none'/>

```

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 25, 2021, 11:28am UTC](https://discuss.elastic.co/t/clean-install-of-es-7-13-2-on-ubuntu-20-04-hangs/276808/8 "2021-07-25T11:28:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
