Not able to start Elasticsearch, no matter what I do

prajwalgmpp · July 12, 2020, 4:24pm

Running Ubuntu 20.04 on Oracle VM.

Deployed two Ubuntu instances on AWS.

I will list out all the steps I did. Please let me know where I have gone wrong.

Copied the pem file to home directory cp ~/Downloads/TetraSearchDemo.pem
Changed the permissions of the pem file to -r-------- 1
Connected to Ubuntu instance ssh -i ".pem file" <public dns of the instance>
Installed Java sudo apt install openjdk-11-jre-headless
Checked the presence of Java and got the following versions

openjdk version "11.0.7" 2020-04-14 OpenJDK Runtime Environment (build 11.0.7+10-post-Ubuntu-2ubuntu218.04) OpenJDK 64-Bit Server VM (build 11.0.7+10-post-Ubuntu-2ubuntu218.04, mixed mode, sharing)

I think this is where something is going wrong. In the training video I am following, the trainer is getting java version "1.7.0_111" along with the above versions but I am not. Please advise if it is ok.

Moving on

Downloaded Debian version of elasticsearc: wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.8.0-amd64.deb
Added the package sudo dpkg -i elasticsearch-7.8.0-amd64.deb
Added some default values sudo update-rc.d elasticsearch defaults 95 10 Here the trainer got an output saying Adding system startup for but I did not get any output.
Finally, tried to start elasticsearch sudo /etc/init.d/elasticsearch start

but continuously getting this error

....] Starting elasticsearch (via systemctl): elasticsearch.serviceJob for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.  failed!

I have followed the tips offered in other threads like editing the nano text file and so on, but nothing worked. Please help me

Edit: I entered the yml file and did the following changes:

Deleted discovery.seed_hosts: ["instance1 private IP", "instance2 private IP"]

and added the following

discovery.zen.ping.unicast.hosts: ["instance1 private IP", "instance2 private IP"] discovery.zen.ping.multicast.enabled: false

After making the above changes, to check if they have taken effect curl -XGET "instance1 private IP:9200" , getting the following error:

curl: (7) Failed to connect to <ip> port 9200: Connection refused

Vinayak_Sapre · July 12, 2020, 10:30pm

@prajwalgmpp
ES 7.8 requires JDK 1.8 or 11.
systemctl status elasticsearch.service will show you that service was not started. So connection refused for curl is expected.

Can you post elasticsearch.yml and output of journalctl -u elasticsearch.service?

Also you should start service using sudo systemctl start elasticsearch.service.

warkolm · July 13, 2020, 1:41am

What's in /var/log/elasticsearch/elasticsearch.log?

prajwalgmpp · July 13, 2020, 1:04pm

Hi @Vinayak_Sapre,

The start is still failing with the below error even with sudo systemctl start elasticsearch.service

`Job for elasticsearch.service failed because the control process exited with error code.`
`See "systemctl status elasticsearch.service" and "journalctl -xe" for details.`

The output for journalctl -u elasticsearch.service is

Jul 13 12:17:29 ip-172-31-6-123 systemd[1]: Starting Elasticsearch...
Jul 13 12:17:31 ip-172-31-6-123 systemd-entrypoint[22108]: Exception in thread "main" java.lang.RuntimeException: starting java failed with [1]
Jul 13 12:17:31 ip-172-31-6-123 systemd-entrypoint[22108]: output:
Jul 13 12:17:31 ip-172-31-6-123 systemd-entrypoint[22108]: #
Jul 13 12:17:31 ip-172-31-6-123 systemd-entrypoint[22108]: # There is insufficient memory for the Java Runtime Environment to continue.
Jul 13 12:17:31 ip-172-31-6-123 systemd-entrypoint[22108]: # Native memory allocation (mmap) failed to map 1073741824 bytes for committing reserved memory.
Jul 13 12:17:31 ip-172-31-6-123 systemd-entrypoint[22108]: # An error report file with more information is saved as:
Jul 13 12:17:31 ip-172-31-6-123 systemd-entrypoint[22108]: # /var/log/elasticsearch/hs_err_pid22238.log
Jul 13 12:17:31 ip-172-31-6-123 systemd-entrypoint[22108]: error:
Jul 13 12:17:31 ip-172-31-6-123 systemd-entrypoint[22108]: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c0000000, 1073741824, 0) failed; error='Not enough space' (errno=12)
Jul 13 12:17:31 ip-172-31-6-123 systemd-entrypoint[22108]:         at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:126)
Jul 13 12:17:31 ip-172-31-6-123 systemd-entrypoint[22108]:         at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:88)
Jul 13 12:17:31 ip-172-31-6-123 systemd-entrypoint[22108]:         at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59)
Jul 13 12:17:31 ip-172-31-6-123 systemd-entrypoint[22108]:         at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:137)
Jul 13 12:17:31 ip-172-31-6-123 systemd-entrypoint[22108]:         at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:95)
Jul 13 12:17:31 ip-172-31-6-123 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Jul 13 12:17:31 ip-172-31-6-123 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Jul 13 12:17:31 ip-172-31-6-123 systemd[1]: Failed to start Elasticsearch.

It appears that there is insufficient memory, but in /etc/elasticsearch/jvm.options it is showing as
-Xms1g
-Xmx1g
which is normal right?
Please advise.

prajwalgmpp · July 13, 2020, 1:10pm

There was no file called elasticsearch.log in elasticsearch directory. However:

var/log/elasticsearch# ls
gc.log     gc.log.01  gc.log.03  gc.log.05  hs_err_pid15167.log  hs_err_pid15837.log  hs_err_pid21806.log  hs_err_pid22238.log
gc.log.00  gc.log.02  gc.log.04  gc.log.06  hs_err_pid15632.log  hs_err_pid1997.log   hs_err_pid22074.log  hs_err_pid2967.log

root@ip-1xx.xx.xxx:/var/log/elasticsearch# tail hs_err_pid2967.log
    CPU:total 1 (initial active 1) (1 cores per cpu, 1 threads per core) family 6 model 63 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, lzcnt, tsc, bmi1, bmi2, fma
    CPU Model and flags from /proc/cpuinfo:
    model name	: Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
    flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault invpcid_single pti fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt

Memory: 4k page, physical 1007296k(133812k free), swap 0k(0k free)

vm_info: OpenJDK 64-Bit Server VM (14.0.1+7) for linux-amd64 JRE (14.0.1+7), built on Apr 15 2020 15:16:40 by "jenkins" with gcc 7.5.0

END.

Please advise further

Vinayak_Sapre · July 13, 2020, 2:50pm

@prajwalgmpp

Looks like you have <1GB physical memory and ~130MB is free.

Which AWS instance type are you using?

prajwalgmpp · July 15, 2020, 10:45am

I am using t2.micro instance (free tier eligibile)

Since my above approach was not working, I decided to follow the steps given in: Get Elasticsearch up and running | Elasticsearch Guide [8.11] | Elastic

After following the steps and starting elasticsearch, the terminal is frozen at the following message:

 [2020-07-15T16:08:58,583][INFO ][o.e.c.s.ClusterApplierService] [prajwal-VirtualBox] master node changed {previous [], current [{prajwal-VirtualBox}{yWtRIWh2SkevCle1NKWKHQ}{2VxmjHSkTcqd8PTC_4KNVQ}{127.0.0.1}{127.0.0.1:9300}{dilmrt}{ml.machine_memory=3137728512, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}, term: 5, version: 31, reason: Publication{term=5, version=31}
    [2020-07-15T16:08:58,654][INFO ][o.e.h.AbstractHttpServerTransport] [prajwal-VirtualBox] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
    [2020-07-15T16:08:58,654][INFO ][o.e.n.Node               ] [prajwal-VirtualBox] started
    [2020-07-15T16:08:59,023][INFO ][o.e.l.LicenseService     ] [prajwal-VirtualBox] license [ed7a5d7a-dd8e-4889-82b7-109329ee1b6d] mode [basic] - valid
    [2020-07-15T16:08:59,024][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [prajwal-VirtualBox] Active license is now [BASIC]; Security is disabled
    [2020-07-15T16:08:59,037][INFO ][o.e.g.GatewayService     ] [prajwal-VirtualBox] recovered [0] indices into cluster_state

Please advise.

dadoonet · July 15, 2020, 2:15pm

It looks good and started...

system · August 12, 2020, 2:15pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.