Self hosted elasticsearch on AWS ec2 server

I'm new to Elasticsearch, and I'm trying to install and run Elasticsearch on a CentOS 8 server in AWS. Currently it is a small server (1GB memory, 15GB SSD, 1 CPU)

I have installed the software, but when trying to run the service it times out.

[ec2-user@ip-172-31-43-130 etc]$ sudo systemctl start elasticsearch.service
Job for elasticsearch.service failed because a timeout was exceeded.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

[ec2-user@ip-172-31-43-130 etc]$ journalctl -xe
Nov 29 19:08:19 ip-172-31-43-130.us-west-2.compute.internal NetworkManager[677]: <info>  [1638212899.3863] dhcp4 (eth0)>
Nov 29 19:08:19 ip-172-31-43-130.us-west-2.compute.internal NetworkManager[677]: <info>  [1638212899.3874] dhcp4 (eth0)>
Nov 29 19:08:19 ip-172-31-43-130.us-west-2.compute.internal NetworkManager[677]: <info>  [1638212899.3874] dhcp4 (eth0)>
Nov 29 19:08:19 ip-172-31-43-130.us-west-2.compute.internal NetworkManager[677]: <info>  [1638212899.3874] dhcp4 (eth0)>
Nov 29 19:08:19 ip-172-31-43-130.us-west-2.compute.internal NetworkManager[677]: <info>  [1638212899.3874] dhcp4 (eth0)>
Nov 29 19:08:19 ip-172-31-43-130.us-west-2.compute.internal NetworkManager[677]: <info>  [1638212899.3874] dhcp4 (eth0)>
Nov 29 19:08:19 ip-172-31-43-130.us-west-2.compute.internal NetworkManager[677]: <info>  [1638212899.3874] dhcp4 (eth0)>
Nov 29 19:08:19 ip-172-31-43-130.us-west-2.compute.internal dbus-daemon[478]: [system] Activating via systemd: service >
Nov 29 19:08:19 ip-172-31-43-130.us-west-2.compute.internal systemd[1]: Starting Network Manager Script Dispatcher Serv>
-- Subject: Unit NetworkManager-dispatcher.service has begun start-up
-- Defined-By: systemd
-- Support: https://support.oracle.com
--
-- Unit NetworkManager-dispatcher.service has begun starting up.
Nov 29 19:08:19 ip-172-31-43-130.us-west-2.compute.internal dbus-daemon[478]: [system] Successfully activated service '>
Nov 29 19:08:19 ip-172-31-43-130.us-west-2.compute.internal systemd[1]: Started Network Manager Script Dispatcher Servi>
-- Subject: Unit NetworkManager-dispatcher.service has finished start-up
-- Defined-By: systemd
-- Support: https://support.oracle.com
--
-- Unit NetworkManager-dispatcher.service has finished starting up.
--
-- The start-up result is done.
Nov 29 19:08:29 ip-172-31-43-130.us-west-2.compute.internal systemd[1]: NetworkManager-dispatcher.service: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: https://support.oracle.com
--
-- The unit NetworkManager-dispatcher.service has successfully entered the 'dead' state.

[ec2-user@ip-172-31-43-130 etc]$ sudo systemctl start elasticsearch.service
Job for elasticsearch.service failed because a timeout was exceeded.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
[ec2-user@ip-172-31-43-130 etc]$ journalctl -xe
-- Defined-By: systemd
-- Support: https://support.oracle.com
--
-- Unit elasticsearch.service has begun starting up.
Nov 29 19:15:40 ip-172-31-43-130.us-west-2.compute.internal systemd-entrypoint[2160]: WARNING: A terminally deprecated >
Nov 29 19:15:40 ip-172-31-43-130.us-west-2.compute.internal systemd-entrypoint[2160]: WARNING: System::setSecurityManag>
Nov 29 19:15:40 ip-172-31-43-130.us-west-2.compute.internal systemd-entrypoint[2160]: WARNING: Please consider reportin>
Nov 29 19:15:40 ip-172-31-43-130.us-west-2.compute.internal systemd-entrypoint[2160]: WARNING: System::setSecurityManag>
Nov 29 19:15:44 ip-172-31-43-130.us-west-2.compute.internal systemd-entrypoint[2160]: WARNING: A terminally deprecated >
Nov 29 19:15:44 ip-172-31-43-130.us-west-2.compute.internal systemd-entrypoint[2160]: WARNING: System::setSecurityManag>
Nov 29 19:15:44 ip-172-31-43-130.us-west-2.compute.internal systemd-entrypoint[2160]: WARNING: Please consider reportin>
Nov 29 19:15:44 ip-172-31-43-130.us-west-2.compute.internal systemd-entrypoint[2160]: WARNING: System::setSecurityManag>
Nov 29 19:16:31 ip-172-31-43-130.us-west-2.compute.internal sshd[2332]: error: kex_exchange_identification: Connection >
Nov 29 19:16:51 ip-172-31-43-130.us-west-2.compute.internal systemd[1]: elasticsearch.service: start operation timed ou>
Nov 29 19:17:29 ip-172-31-43-130.us-west-2.compute.internal systemd[1]: elasticsearch.service: Failed with result 'time>
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://support.oracle.com
--
-- The unit elasticsearch.service has entered the 'failed' state with result 'timeout'.
Nov 29 19:17:29 ip-172-31-43-130.us-west-2.compute.internal systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: https://support.oracle.com
--
-- Unit elasticsearch.service has failed.
--
-- The result is failed.
Nov 29 19:17:30 ip-172-31-43-130.us-west-2.compute.internal sudo[2157]: pam_unix(sudo:session): session closed for user

[ec2-user@ip-172-31-43-130 etc]$ systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: timeout) since Mon 2021-11-29 19:17:29 GMT; 8min ago
     Docs: https://www.elastic.co
  Process: 2160 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code>
 Main PID: 2160 (code=exited, status=143)

Nov 29 19:15:40 ip-172-31-43-130.us-west-2.compute.internal systemd-entrypoint[2160]: WARNING: System::setSecurityManag>
Nov 29 19:15:40 ip-172-31-43-130.us-west-2.compute.internal systemd-entrypoint[2160]: WARNING: Please consider reportin>
Nov 29 19:15:40 ip-172-31-43-130.us-west-2.compute.internal systemd-entrypoint[2160]: WARNING: System::setSecurityManag>
Nov 29 19:15:44 ip-172-31-43-130.us-west-2.compute.internal systemd-entrypoint[2160]: WARNING: A terminally deprecated >
Nov 29 19:15:44 ip-172-31-43-130.us-west-2.compute.internal systemd-entrypoint[2160]: WARNING: System::setSecurityManag>
Nov 29 19:15:44 ip-172-31-43-130.us-west-2.compute.internal systemd-entrypoint[2160]: WARNING: Please consider reportin>
Nov 29 19:15:44 ip-172-31-43-130.us-west-2.compute.internal systemd-entrypoint[2160]: WARNING: System::setSecurityManag>
Nov 29 19:16:51 ip-172-31-43-130.us-west-2.compute.internal systemd[1]: elasticsearch.service: start operation timed ou>
Nov 29 19:17:29 ip-172-31-43-130.us-west-2.compute.internal systemd[1]: elasticsearch.service: Failed with result 'time>
Nov 29 19:17:29 ip-172-31-43-130.us-west-2.compute.internal systemd[1]: Failed to start Elasticsearch.

googling for solutions has so far left me without any concrete answers. I may just not be googling for the right terms.

You need to check in the systems log, probably on /var/log/messages, there you will find why the server is not starting up.

What is the version you are using? Also, I would say that 1 GB is too small for Elasticsearch.

/var/log/messages doesn't tell me anything new

Nov 29 19:49:09 ip-172-31-43-130 systemd[1]: Starting Elasticsearch...
Nov 29 19:49:13 ip-172-31-43-130 systemd-entrypoint[2384]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 29 19:49:13 ip-172-31-43-130 systemd-entrypoint[2384]: WARNING: System::setSecurityManager has been called by org.elasticsearch.bootstrap.Elasticsearch (file:/usr/share/elasticsearch/lib/elasticsearch-7.15.2.jar)
Nov 29 19:49:13 ip-172-31-43-130 systemd-entrypoint[2384]: WARNING: Please consider reporting this to the maintainers of org.elasticsearch.bootstrap.Elasticsearch
Nov 29 19:49:13 ip-172-31-43-130 systemd-entrypoint[2384]: WARNING: System::setSecurityManager will be removed in a future release
Nov 29 19:49:17 ip-172-31-43-130 systemd-entrypoint[2384]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 29 19:49:17 ip-172-31-43-130 systemd-entrypoint[2384]: WARNING: System::setSecurityManager has been called by org.elasticsearch.bootstrap.Security (file:/usr/share/elasticsearch/lib/elasticsearch-7.15.2.jar)
Nov 29 19:49:17 ip-172-31-43-130 systemd-entrypoint[2384]: WARNING: Please consider reporting this to the maintainers of org.elasticsearch.bootstrap.Security
Nov 29 19:49:17 ip-172-31-43-130 systemd-entrypoint[2384]: WARNING: System::setSecurityManager will be removed in a future release
Nov 29 19:50:24 ip-172-31-43-130 systemd[1]: elasticsearch.service: start operation timed out. Terminating.
Nov 29 19:51:08 ip-172-31-43-130 systemd[1]: elasticsearch.service: Failed with result 'timeout'.
Nov 29 19:51:08 ip-172-31-43-130 systemd[1]: Failed to start Elasticsearch.

As for the 1GB memory, I can always increase it once I have things up and running. Do you think that needs to be larger just to get it up and running (and start feeding it logs with 1 instance of filebeat feeding data into it)?

Is this everything you have in /var/log/messages? It should have more lines saying why the service failed. Try to start it again and look in the log for any hints.

Also, do you have anything in Elasticsearch logs? Per default they would be in /var/log/elasticsearch.

Share your elasticsearch.yml as well. Which is the version that you are using?

I increased my memory to 4GB, and have been able to get it running (after a couple of other .yml config issues). Thanks for the help.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.