Error running kibana 8.14 in AL 2023 ARM

Seeing an error during the install:

/usr/share/kibana/bin/kibana-keystore: line 29: 374498 Illegal instruction     (core dumped) NODE_OPTIONS="$KBN_NODE_OPTS $NODE_OPTIONS" NODE_ENV=production "${NODE}" "${DIR}/src/cli_keystore/dist" "$@"
chown: cannot access '/etc/kibana/kibana.keystore': No such file or directory
chmod: cannot access '/etc/kibana/kibana.keystore': No such file or directory
warning: %posttrans(kibana-8.14.3-1.aarch64) scriptlet failed, exit status 1

Kibana service status shows:

kibana.service: Failed with result 'core-dump'.

The same error is seen when using kibana 8.17 too. Has anyone else seen the same issue ?

Welcome to the community @Tara_John

A little bit more info please, what/how are you downloading/installing, which specific version of AL are you running and how and where, ... ?

Were you following the official docs, or a 3rd party guide, or .. ?

Outputs of "uname -a" and "cat /etc/*release" would be start, plus the commands you used to do the installation.

Thanks for the prompt reply !

I am following the official guide : Install Kibana with RPM | Kibana Guide [8.14] | Elastic
I was previously running kibana in AL2 x86_64 arch. But we are moving to AL 2023 ARM. Running them in ec2 instances.

Output of uname -a

Linux ip-XXXXXXX 6.1.119-129.201.amzn2023.aarch64 #1 SMP Tue Dec  3 21:06:52 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
cat /etc/*release

Amazon Linux release 2023.6.20250114 (Amazon Linux)
NAME="Amazon Linux"
VERSION="2023"
ID="amzn"
ID_LIKE="fedora"
VERSION_ID="2023"
PLATFORM_ID="platform:al2023"
PRETTY_NAME="Amazon Linux 2023.6.20250114"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2023"
HOME_URL="https://aws.amazon.com/linux/amazon-linux-2023/"
DOCUMENTATION_URL="https://docs.aws.amazon.com/linux/"
SUPPORT_URL="https://aws.amazon.com/premiumsupport/"
BUG_REPORT_URL="https://github.com/amazonlinux/amazon-linux-2023"
VENDOR_NAME="AWS"
VENDOR_URL="https://aws.amazon.com/"
SUPPORT_END="2028-03-15"
Amazon Linux release 2023.6.20250114 (Amazon Linux)

Since this is just an image upgrade, we are keeping the kibana version same as previous setup : 8.14.3

Installation command : `sudo dnf install kibana-8.14.3`

Thanks for sharing the info.

Is this on a completely fresh EC2 instance? Which specific ami are you using, particularly is it somehow "homemade"?

Not sure if this helps, but I just tried with docker image of amazonlinux:latest , which is a little bit newer

PRETTY_NAME="Amazon Linux 2023.6.20250128"

with kernel

Linux 064fa1a465fc 6.12.5-linuxkit #1 SMP Tue Jan 21 10:23:32 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux

and the kibana install of 8.14.3 just worked.

I only added the repo and imported key, installed a couple of helper packages (findutils and shadow-utils) and then got

# cat /etc/yum.repos.d/kib.repo
[kibana-8.x]
name=Kibana repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

# dnf install kibana-8.14.3
Kibana repository for 8.x packages                                                       28 MB/s |  89 MB     00:03
Last metadata expiration check: 0:00:15 ago on Mon Feb 10 20:54:21 2025.
Dependencies resolved.
========================================================================================================================
 Package                   Architecture               Version                       Repository                     Size
========================================================================================================================
Installing:
 kibana                    aarch64                    8.14.3-1                      kibana-8.x                    339 M

Transaction Summary
========================================================================================================================
Install  1 Package

Total download size: 339 M
Installed size: 1.0 G
Is this ok [y/N]: y
Downloading Packages:
kibana-8.14.3-aarch64.rpm                                                                22 MB/s | 339 MB     00:15
------------------------------------------------------------------------------------------------------------------------
Total                                                                                    22 MB/s | 339 MB     00:15
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                1/1
  Running scriptlet: kibana-8.14.3-1.aarch64                                                                        1/1
  Installing       : kibana-8.14.3-1.aarch64                                                                        1/1
  Running scriptlet: kibana-8.14.3-1.aarch64                                                                        1/1
Creating kibana group... OK
Creating kibana user... OK

Kibana is currently running with legacy OpenSSL providers enabled! For details and instructions on how to disable see https://www.elastic.co/guide/en/kibana/8.14/production.html#openssl-legacy-provider
Created Kibana keystore in /etc/kibana/kibana.keystore

  Verifying        : kibana-8.14.3-1.aarch64                                                                        1/1

Installed:
  kibana-8.14.3-1.aarch64

Complete!

# cat /etc/*release
Amazon Linux release 2023.6.20250128 (Amazon Linux)
NAME="Amazon Linux"
VERSION="2023"
ID="amzn"
ID_LIKE="fedora"
VERSION_ID="2023"
PLATFORM_ID="platform:al2023"
PRETTY_NAME="Amazon Linux 2023.6.20250128"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2023"
HOME_URL="https://aws.amazon.com/linux/amazon-linux-2023/"
DOCUMENTATION_URL="https://docs.aws.amazon.com/linux/"
SUPPORT_URL="https://aws.amazon.com/premiumsupport/"
BUG_REPORT_URL="https://github.com/amazonlinux/amazon-linux-2023"
VENDOR_NAME="AWS"
VENDOR_URL="https://aws.amazon.com/"
SUPPORT_END="2029-06-30"
Amazon Linux release 2023.6.20250128 (Amazon Linux)

# uname -a
Linux 064fa1a465fc 6.12.5-linuxkit #1 SMP Tue Jan 21 10:23:32 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux

Your error:

/usr/share/kibana/bin/kibana-keystore: line 29: 374498 Illegal instruction     (core dumped) NODE_OPTIONS="$KBN_NODE_OPTS $NODE_OPTIONS" NODE_ENV=production "${NODE}" "${DIR}/src/cli_keystore/dist" "$@"

would suggest some mismatch between arch somewhere.

Here's the MD5 of the file I installed (used --downloadonly to just download it)

# md5sum /var/cache/dnf/kibana-8.x-12170fef1b21a4ec/packages/kibana-8.14.3-aarch64.rpm
ff2e746f975bd00b190b0d226f590351  /var/cache/dnf/kibana-8.x-12170fef1b21a4ec/packages/kibana-8.14.3-aarch64.rpm

Yes this is a completely fresh EC2 instance and the AMI is an in-house image with 2023.6.20250114 being the base. We won't be able to use the latest image.

Seems like everything except the image version is same from both the setup. Package name kibana-8.14.3-1.aarch64

Seems like everything except the image version is same

Not quire, I'm on docker, you're on EC2. I'm using M1 Mac, who-knows-what hardware is being used by AWS, but I dont think its a M1 Mac mini, ...

I dunno if it's a possibility, but could you test with a non-in-house image? If it works there, then ... something not right in the image would be my guess.

Also, just curious really, what does "cat /proc/cpuinfo | sort -u" get you?

My docker is running on my M1 Mac, and I get

# cat /proc/cpuinfo | sort -u

BogoMIPS	: 48.00
CPU architecture: 8
CPU implementer	: 0x61
CPU part	: 0x000
CPU revision	: 0
CPU variant	: 0x0
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint
processor	: 0
processor	: 1
processor	: 2
processor	: 3
processor	: 4
processor	: 5
processor	: 6
processor	: 7
# cat /proc/cpuinfo | sort -u

BogoMIPS	: 2000.00
CPU architecture: 8
CPU implementer	: 0x41
CPU part	: 0xd4f
CPU revision	: 1
CPU variant	: 0x0
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
processor	: 0
processor	: 1

Tried docker in my mac with amazon linux 2023 and was able to come past the kibana.keystore error. So it might be something with the in-house image.
I am using a c8g instance type, Arm-based AWS Graviton4 processor, that seems to satisfy AL2023 ARM requirements.

So, if you spin up a random c8g EC2 instance type on EC2, you can try just installing kibana there. If that works, it's something in the prep of your own ami. If not, then I'd open a case with AWS. Also perhaps your in-house image has some hardening or optimizing that, er, is not ideal here.

Just in passing, I'm sure the c8g instance type is great, but it's certainly not required for kibana. AWS market as:

"Amazon EC2 C8g instances are ideal for compute-intensive workloads, such as high performance computing (HPC), batch processing, gaming, video encoding, scientific modeling, distributed analytics, CPU-based machine learning (ML) inference, and ad serving"

I'm not sure that kibana is taking most advantage of that arch, though I'm happy to be corrected.

I did the quick test above, spun up a c8g.medium EC2 instance, and get same error as you with 8.14.3, so its not your in-house created ami.

Ah, so as the problem is just with the bundled node:

# /usr/share/kibana/node/bin/node
Illegal instruction (core dumped)

so I tried with the latest kibana release (8.17.2) and node there works OK, as does kibana in my very brief testing.

Maybe you were just unlucky with 8.14.3 build, in 8.17.3 its moved around a bit and is now at

# /usr/share/kibana/node/glibc-217/bin/node
Welcome to Node.js v20.18.2.
Type ".help" for more information.

# uname -a
Linux ip-172-31-10-19.eu-central-1.compute.internal 6.1.127-135.201.amzn2023.aarch64 #1 SMP Tue Jan 28 23:19:27 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux

I guess if you have Elastic support contract you might be able to get them to release a patch/fix for 8.14.3 ? Otherwise, an upgrade should resolve it.

is likely the bug

Thanks for all the help !

We will move forward with using the latest version !