Elastic failing on Ubuntu 18 on AWS

Elastic Stack Elasticsearch
1

I've installed Elastic 7.5 on a t3a.small instance via AWS as part of a rebuild of an existing PHP application, also using Elastic on the same type of instance, although this would be Elastic 6.2 or something in the 6.x range.

This new version of the application is built on Node. Vue, Express and so on.

I'm able to get Elastic running for a short time before it fails:

● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: signal) since Sun 2020-02-16 15:57:51 UTC; 1h 18min ago
Docs: http://www.elastic.co
Process: 1672 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=killed, signal=KILL)
Main PID: 1672 (code=killed, signal=KILL)

Feb 16 14:43:05 ip-172-31-14-190 systemd[1]: Starting Elasticsearch...
Feb 16 14:43:06 ip-172-31-14-190 elasticsearch[1672]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Feb 16 14:43:55 ip-172-31-14-190 systemd[1]: Started Elasticsearch.
Feb 16 15:57:51 ip-172-31-14-190 systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
Feb 16 15:57:51 ip-172-31-14-190 systemd[1]: elasticsearch.service: Failed with result 'signal'.

Is this a memory issue?

I've attempted accessing the logs:

sudo nano /var/log/elasticsearch/elasticsearch.log

… but I'm getting permission errors.

2

I think this means Elasticsearch was killed abruptly by an external process, often the kernel's OOM killer. If so, check the kernel logs using dmesg, and also check that you have not set your heap size to greater than 50% of the available RAM.

This needs addressing. You will need access to the logs for a variety of reasons.

3

Hi @DavidTurner, I'll investigate the heap size.

As for the logs, they belong to elasticsearch, and I didn't want to mess around with the ownership without knowing what would happen if I did.

As an example:

ubuntu@ip-172-31-14-190:~$ tail -f /var/log/elasticsearch/elasticsearch.log
tail: cannot open '/var/log/elasticsearch/elasticsearch.log' for reading: Permission denied

ubuntu@ip-172-31-14-190:~$ cat /etc/elasticsearch/elasticsearch.yml
cat: /etc/elasticsearch/elasticsearch.yml: Permission denied

I get the same or similar messages when I use sudo.

So wherever Elastic has resources, they're locked down.

4

A t3a.small instance only has 2GB of RAM. What have you set the heap size to? What else is running on thst host?

1 Like
5

Hi @Christian_Dahlqvist, up until the response by @DavidTurner, I knew nothing of the heap size, so it's whatever the default is.

Also (as explained in the last message) since the installation has used its own user for its resources, I'd prefer to not go changing ownership until I know it won't cause problems.

I'm running Ubuntu 18, and an application built on Vue, Express, Node, MySQL, Redis and so on, with PM2 to manage the restarting of things. I'm also running Kibana in the background, although I suppose that should be disabled.

As I said, I've been running the same instance since with a PHP application using Elastic 6.2 since 2014 without problems.

6

I don't think this is Elasticsearch's doing; at least if it is, it sounds like a bug. An unreadable log file isn't any use to anyone.

I can't think of a mechanism that would forbid root from reading these files. Are they on a local filesystem or is something like NFS involved perhaps?

Note also that if it is the OOM killer then it's the kernel logs you need to check.

7

Hi @DavidTurner, as I said, I made no deviation from the installation instructions, and the user for the Elastic resources is elasticsearch.

As stated, Elastic was installed onto Ubuntu 18 LTS on AWS — a clean AMI, at the time.

An image of: "cd /var/log | ls -la":

Screen Shot 2020-02-17 at 09.58.47
Screen Shot 2020-02-17 at 09.58.47992×74 20.7 KB

8

My directory listing looks like yours but my root can read the logs just fine:

Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-1057-aws x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Mon Feb 17 11:10:46 UTC 2020

  System load:  0.13              Processes:           109
  Usage of /:   13.6% of 7.69GB   Users logged in:     0
  Memory usage: 2%                IP address for ens5: 10.77.18.48
  Swap usage:   0%

0 packages can be updated.
0 updates are security updates.



The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@ip-10-77-18-48:~$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
OK
ubuntu@ip-10-77-18-48:~$ sudo apt-get install apt-transport-https
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  apt-transport-https
0 upgraded, 1 newly installed, 0 to remove and 45 not upgraded.
Need to get 1692 B of archives.
After this operation, 153 kB of additional disk space will be used.
Get:1 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 apt-transport-https all 1.6.12 [1692 B]
Fetched 1692 B in 0s (0 B/s)
Selecting previously unselected package apt-transport-https.
(Reading database ... 56554 files and directories currently installed.)
Preparing to unpack .../apt-transport-https_1.6.12_all.deb ...
Unpacking apt-transport-https (1.6.12) ...
Setting up apt-transport-https (1.6.12) ...
ubuntu@ip-10-77-18-48:~$ echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
deb https://artifacts.elastic.co/packages/7.x/apt stable main
ubuntu@ip-10-77-18-48:~$ sudo apt-get update && sudo apt-get install elasticsearch
Hit:1 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:3 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu bionic-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu bionic-security InRelease
Get:5 https://artifacts.elastic.co/packages/7.x/apt stable InRelease [7124 B]
Get:6 https://artifacts.elastic.co/packages/7.x/apt stable/main amd64 Packages [28.9 kB]
Fetched 36.0 kB in 0s (99.5 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  elasticsearch
0 upgraded, 1 newly installed, 0 to remove and 45 not upgraded.
Need to get 296 MB of archives.
After this operation, 498 MB of additional disk space will be used.
Get:1 https://artifacts.elastic.co/packages/7.x/apt stable/main amd64 elasticsearch amd64 7.6.0 [296 MB]
Fetched 296 MB in 6s (51.1 MB/s)
Selecting previously unselected package elasticsearch.
(Reading database ... 56558 files and directories currently installed.)
Preparing to unpack .../elasticsearch_7.6.0_amd64.deb ...
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Unpacking elasticsearch (7.6.0) ...
Setting up elasticsearch (7.6.0) ...
Created elasticsearch keystore in /etc/elasticsearch
Processing triggers for ureadahead (0.100.0-21) ...
Processing triggers for systemd (237-3ubuntu10.33) ...
ubuntu@ip-10-77-18-48:~$ sudo service elasticsearch start
ubuntu@ip-10-77-18-48:~$ ls -al /var/log | grep elasticsearch
drwxr-s---   2 elasticsearch elasticsearch     4096 Feb 17 11:12 elasticsearch
ubuntu@ip-10-77-18-48:~$ ls -al /var/log/elasticsearch/
ls: cannot open directory '/var/log/elasticsearch/': Permission denied
ubuntu@ip-10-77-18-48:~$ sudo ls -al /var/log/elasticsearch/ | head -n4
total 208
drwxr-s---  2 elasticsearch elasticsearch   4096 Feb 17 11:12 .
drwxrwxr-x 10 root          syslog          4096 Feb 17 11:12 ..
-rw-r--r--  1 elasticsearch elasticsearch  11480 Feb 17 11:12 elasticsearch.log
ubuntu@ip-10-77-18-48:~$ sudo tail /var/log/elasticsearch/elasticsearch.log
[2020-02-17T11:12:43,396][INFO ][o.e.c.m.MetaDataIndexTemplateService] [ip-10-77-18-48] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-7-*]
[2020-02-17T11:12:43,492][INFO ][o.e.c.m.MetaDataIndexTemplateService] [ip-10-77-18-48] adding template [.monitoring-es] for index patterns [.monitoring-es-7-*]
[2020-02-17T11:12:43,559][INFO ][o.e.c.m.MetaDataIndexTemplateService] [ip-10-77-18-48] adding template [.monitoring-beats] for index patterns [.monitoring-beats-7-*]
[2020-02-17T11:12:43,614][INFO ][o.e.c.m.MetaDataIndexTemplateService] [ip-10-77-18-48] adding template [.monitoring-alerts-7] for index patterns [.monitoring-alerts-7]
[2020-02-17T11:12:43,664][INFO ][o.e.c.m.MetaDataIndexTemplateService] [ip-10-77-18-48] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-7-*]
[2020-02-17T11:12:43,703][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [ip-10-77-18-48] adding index lifecycle policy [watch-history-ilm-policy]
[2020-02-17T11:12:43,765][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [ip-10-77-18-48] adding index lifecycle policy [ilm-history-ilm-policy]
[2020-02-17T11:12:43,816][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [ip-10-77-18-48] adding index lifecycle policy [slm-history-ilm-policy]
[2020-02-17T11:12:44,037][INFO ][o.e.l.LicenseService     ] [ip-10-77-18-48] license [fdbaf428-9b46-482d-894f-abf12ce45fbc] mode [basic] - valid
[2020-02-17T11:12:44,038][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [ip-10-77-18-48] Active license is now [BASIC]; Security is disabled
ubuntu@ip-10-77-18-48:~$
9

Hi @DavidTurner, technicalities aside, what options do I have? I need to get Elastic working in this environment.

10

I'm not sure what you mean by "technicalities". There is no question that you need to be able to see your server logs. If you can't pinpoint why things like sudo tail /var/log/elasticsearch/elasticsearch.log don't work for you then you'll need to call in some general sysadmin help, as this is outside the scope of what we can cover on this forum. Without logs, you will find it nearly impossible to do any kind of troubleshooting.

I'm also not sure what's wrong with the options described in this thread already. As I said above:

Have you done so? Did you see evidence that the OOM killer was killing Elasticsearch?

Also, as I said above:

If you're running on a tiny 2GB machine then you must account for some overhead (the kernel, other services on the same machine, etc.) and reduce your node's heap size from the default of 1GB. Have you done so? Did you see any improvement? How did this change the logs?

11

Hi @DavidTurner, since the Elastic installation is responsible for what's happened, I fail to see how this could be outside of the scope of this forum — what better definition of a support need is there when the software fails to complete its intended task?

As explained, I'm using the exact same instance as an existing product that uses PHP and a stock installation of Elastic 6.5 that's run continuous since 2014.

I'll look into the OOM issue, but this is taking me far beyond Elastic and into the realms of the OS itself, which is far outside the scope of this forum.

As explained, since Elastic has imposed itself as the user for its resources, I have zero access to the files that control the heap size.

12

Ok, let's take this one step at a time. If you run sudo tail /var/log/elasticsearch/elasticsearch.log what do you get?

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.