Elasticsearch installation

It may be issue with the port 9200. You can check to see if it can be connected with the command nc:

nc -z -v localhost 9200

If that test is successfully connected, then maybe the java virtual machine (jvm) encountered some problem. Check for the java process:

ps -ef | grep java

If that java process is running with no issue, then it may have problem listen to port 9200. Check and you should see something like following:

netstat -tulpn | grep 9200
tcp6       0      0 <host IP>:9200      :::*                    LISTEN      <PID>/java

$ sudo cat /etc/Elasticsearch/jvm.options
################################################################

JVM configuration

################################################################

WARNING: DO NOT EDIT THIS FILE. If you want to override the

JVM options in this file, or set any additional options, you

should create one or more files in the jvm.options.d

directory containing your adjustments.

See Settng JVM options | Elasticsearch Guide [7.15] | Elastic

for more information.

################################################################

################################################################

IMPORTANT: JVM heap size

################################################################

The heap size is automatically configured by Elasticsearch

based on the available memory in your system and the roles

each node is configured to fulfill. If specifying heap is

required, it should be done through a file in jvm.options.d,

and the min and max should be set to the same value. For

example, to set the heap to 4 GB, create a new file in the

jvm.options.d directory containing these lines:

-Xms4g

-Xmx4g

See Heap size settings | Elasticsearch Guide [7.15] | Elastic

for more information

################################################################

################################################################

Expert settings

################################################################

All settings below here are considered expert settings. Do

not adjust them unless you understand what you are doing. Do

not edit them in this file; instead, create a new file in the

jvm.options.d directory containing your adjustments.

################################################################

GC configuration

8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly

G1GC Configuration

NOTE: G1 GC is only supported on JDK version 10 or later

to use G1GC, uncomment the next two lines and update the version on the

following three lines to your version of the JDK

10-13:-XX:-UseConcMarkSweepGC

10-13:-XX:-UseCMSInitiatingOccupancyOnly

14-:-XX:+UseG1GC

JVM temporary directory

-Djava.io.tmpdir=${ES_TMPDIR}

heap dumps

generate a heap dump when an allocation from the Java heap fails; heap dumps

are created in the working directory of the JVM unless an alternative path is

specified

-XX:+HeapDumpOnOutOfMemoryError

specify an alternative path for heap dumps; ensure the directory exists and

has sufficient space

-XX:HeapDumpPath=/var/lib/Elasticsearch

specify an alternative path for JVM fatal error logs

-XX:ErrorFile=/var/log/Elasticsearch/hs_err_pid%p.log

JDK 8 GC logging

8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/Elasticsearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m

JDK 9+ GC logging

9-:-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/Elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m
eugene@eooubuserver:~$

$ nc -z -v localhost 9200
nc: connect to localhost port 9200 (tcp) failed: Connection refused
~$ ps -ef | grep java
logstash    1163       1  1 11:40 ?        00:05:22 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -Djruby.regexp.interruptible=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Dlog4j2.isThreadContextMapInheritable=true -cp /usr/share/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/checker-compat-qual-2.0.0.jar:/usr/share/logstash/logstash-core/lib/jars/commons-codec-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/commons-compiler-3.1.0.jar:/usr/share/logstash/logstash-core/lib/jars/commons-logging-1.2.jar:/usr/share/logstash/logstash-core/lib/jars/error_prone_annotations-2.1.3.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/guava-24.1.1-jre.jar:/usr/share/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-annotations-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-databind-2.9.10.8.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-yaml-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.1.0.jar:/usr/share/logstash/logstash-core/lib/jars/javassist-3.26.0-GA.jar:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.19.0.jar:/usr/share/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-1.2-api-2.14.0.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.14.0.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.14.0.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-jcl-2.14.0.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.14.0.jar:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/reflections-0.9.11.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.30.jar:/usr/share/logstash/logstash-core/lib/jars/snakeyaml-1.23.jar org.logstash.Logstash --path.settings /etc/logstash
eugene   3441475 2663451  0 17:09 pts/0    00:00:00 grep --color=auto java
eugene@eooubuserver:~$ 


$ netstat -tulpn | grep 9200
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
eugene@eooubuserver:~$ 

What address is this ? 0.0.0.0 or 127.0.0.1.... otherwise you might have bind issues the way you are trying to connect

Please look at my /etc/Elasticsearch/Elasticsearch.yml file and see if there is anything I need to change.:

sudo cat /etc/Elasticsearch/Elasticsearch.yml

======================== Elasticsearch Configuration =========================

NOTE: Elasticsearch comes with reasonable defaults for most settings.

Before you set out to tweak and tune the configuration, make sure you

understand what are you trying to accomplish and the consequences.

The primary way of configuring a node is via this file. This template lists

the most important settings you may want to configure for a production cluster.

Please consult the documentation for further information on configuration options:

Elasticsearch Guide | Elastic

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

#cluster.name: my-application

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

#node.name: node-1

Add custom attributes to the node:

#node.attr.rack: r1

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

path.data: /var/lib/Elasticsearch

Path to log files:

path.logs: /var/log/Elasticsearch

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

#bootstrap.memory_lock: true

Make sure that the heap size is set to about half the memory available

on the system and that the owner of the process is allowed to use this

limit.

Elasticsearch performs poorly when the system is swapping the memory.

---------------------------------- Network -----------------------------------

By default Elasticsearch is only accessible on localhost. Set a different

address here to expose this node on the network:

network.host: localhost

By default Elasticsearch listens for HTTP traffic on the first free port it

finds starting at 9200. Set a specific HTTP port here:

http.port: 9200

For more information, consult the network module documentation.

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when this node is started:

The default list of hosts is ["127.0.0.1", "[::1]"]

#discovery.seed_hosts: ["host1", "host2"]

Bootstrap the cluster using an initial set of master-eligible nodes:

#cluster.initial_master_nodes: ["node-1", "node-2"]

For more information, consult the discovery and cluster formation module documentation.

---------------------------------- Various -----------------------------------

Require explicit names when deleting indices:

#action.destructive_requires_name: true
cluster.name: Elasticsearch
node.name: eooubuserver
eugene@eooubuserver:~$

Also , below  is what I have in the tail end of the file /var/log/elasticsearch/elasticsearch.log:
sudo cat /var/log/elasticsearch/elasticsearch.log | tail
[2021-10-06T11:50:31,206][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-shutdown]
[2021-10-06T11:50:31,220][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-sql]
[2021-10-06T11:50:31,221][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-stack]
[2021-10-06T11:50:31,225][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-text-structure]
[2021-10-06T11:50:31,226][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-voting-only-node]
[2021-10-06T11:50:31,226][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-watcher]
[2021-10-06T11:50:31,247][INFO ][o.e.p.PluginsService     ] [eooubuserver] no plugins loaded
[2021-10-06T11:50:32,057][INFO ][o.e.e.NodeEnvironment    ] [eooubuserver] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [183gb], net total_space [219gb], types [ext4]
[2021-10-06T11:50:32,067][INFO ][o.e.e.NodeEnvironment    ] [eooubuserver] heap size [1.4gb], compressed ordinary object pointers [true]
[2021-10-06T11:50:32,417][INFO ][o.e.n.Node               ] [eooubuserver] node name [eooubuserver], node ID [VjrkdT_KQuOXoARgpGfzLg], cluster name [elasticsearch], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
I still continue to get 
"connection refused" :
~$ curl -X GET "http://localhost:9200"
curl: (7) Failed to connect to localhost port 9200: Connection refused

even with 'sudo', I get the same result.

Have you tried actually inputting an ip address and not a name ? This is to rule out binding issues to the localhost or if you have things in your host files messing things up.

Enter the IP Address of the host running elasticsearch to verify the problem.

@zx8086 After changing the word localhost to the host ip address , rebooted the machine and tried to start the es again, this is what I got:

$ sudo systemctl start elasticsearch
[sudo] password for eugene: 
Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
eugene@eooubuserver:~$ 

 and with the 'status' command , this is what I still have :
~$ sudo systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
     Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; ve>
    Drop-In: /etc/systemd/system/elasticsearch.service.d
             └─startup-timeout.conf
     Active: failed (Result: exit-code) since Wed 2021-10-06 13:23:26 EDT; 4min>
       Docs: https://www.elastic.co
    Process: 188930 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint ->
   Main PID: 188930 (code=exited, status=1/FAILURE)

Oct 06 13:23:25 eooubuserver systemd-entrypoint[188930]:         at org.elastic>
Oct 06 13:23:25 eooubuserver systemd-entrypoint[188930]:         at org.elastic>
Oct 06 13:23:25 eooubuserver systemd-entrypoint[188930]:         at org.elastic>
Oct 06 13:23:25 eooubuserver systemd-entrypoint[188930]:         at org.elastic>
Oct 06 13:23:25 eooubuserver systemd-entrypoint[188930]:         at org.elastic>
Oct 06 13:23:25 eooubuserver systemd-entrypoint[188930]:         at org.elastic>
Oct 06 13:23:25 eooubuserver systemd-entrypoint[188930]: For complete error det>
Oct 06 13:23:26 eooubuserver systemd[1]: elasticsearch.service: Main process ex>
Oct 06 13:23:26 eooubuserver systemd[1]: elasticsearch.service: Failed with res>
Oct 06 13:23:26 eooubuserver systemd[1]: Failed to start Elasticsearch.

From your ps -ef | grep java command output, it looks like you have logstash running on the same machine. One problem I had in the past was that I couldn't run elasticsearch and logstash on the same machine. They run fine on two separate machines.

So one suggestion is: Stop logstash and see if elasticsearch can start and run with no problem.

@eugestel are you running with default configurations (jvm.options), no other changes in other files ?

What are the tech specs of the host ?

@zx8086 Yes I am running with ddefault (jvm.options) file. The other file that I made a change is with the Elasticsearch config file (/etc/Elasticsearch/Elasticsearch.yml) and it is the addition of the following:

cluster.name: Elasticsearch
node.name: eooubuserver

I don't understand what is meant by the host 'tech specs'. Please enlighten me. Thanks.

@thanhlam I have stopped and disabled logstash but still Elasticsearch will not start.:

~$ sudo systemctl start elasticsearch.service 
Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xe" for detail
$ sudo systemctl status elasticsearch.service 
● elasticsearch.service - Elasticsearch
     Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; >
    Drop-In: /etc/systemd/system/elasticsearch.service.d
             └─startup-timeout.conf
     Active: failed (Result: exit-code) since Wed 2021-10-06 16:38:35 EDT; 4m>
       Docs: https://www.elastic.co
    Process: 255234 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint>
   Main PID: 255234 (code=exited, status=1/FAILURE)

Oct 06 16:38:34 eooubuserver systemd-entrypoint[255234]:         at org.elast>
Oct 06 16:38:34 eooubuserver systemd-entrypoint[255234]:         at org.elast>
Oct 06 16:38:34 eooubuserver systemd-entrypoint[255234]:         at org.elast>
Oct 06 16:38:34 eooubuserver systemd-entrypoint[255234]:         at org.elast>
Oct 06 16:38:34 eooubuserver systemd-entrypoint[255234]:         at org.elast>
Oct 06 16:38:34 eooubuserver systemd-entrypoint[255234]:         at org.elast>
Oct 06 16:38:34 eooubuserver systemd-entrypoint[255234]: For complete error d>
Oct 06 16:38:35 eooubuserver systemd[1]: elasticsearch.service: Main process >
Oct 06 16:38:35 eooubuserver systemd[1]: elasticsearch.service: Failed with r>
Oct 06 16:38:35 eooubuserver systemd[1]: Failed to start Elasticsearch.
eugene@eooubuserver:~$ 

and

~$ sudo journalctl -xe 
Oct 06 16:45:46 eooubuserver nextcloud.apache[347128]: Nextcloud is not insta>
Oct 06 16:45:46 eooubuserver nextcloud.apache[347128]:                       >
Oct 06 16:45:46 eooubuserver nextcloud.apache[347128]:   There are no command>
Oct 06 16:45:46 eooubuserver nextcloud.apache[347128]:                       >
Oct 06 16:45:47 eooubuserver nextcloud.apache[347194]: Nextcloud is not insta>
Oct 06 16:45:47 eooubuserver nextcloud.apache[347194]:                       >
Oct 06 16:45:47 eooubuserver nextcloud.apache[347194]:   There are no command>
Oct 06 16:45:47 eooubuserver nextcloud.apache[347194]:                       >
Oct 06 16:45:47 eooubuserver nextcloud.apache[347263]: Nextcloud is not insta>
Oct 06 16:45:47 eooubuserver nextcloud.apache[347263]:                       >
Oct 06 16:45:47 eooubuserver nextcloud.apache[347263]:   There are no command>
Oct 06 16:45:47 eooubuserver nextcloud.apache[347263]:                       >
Oct 06 16:45:47 eooubuserver sudo[347306]:   eugene : TTY=pts/1 ; PWD=/home/e>
Oct 06 16:45:47 eooubuserver sudo[347306]: pam_unix(sudo:session): session op>
Oct 06 16:45:47 eooubuserver nextcloud.apache[347331]: Nextcloud is not insta>
Oct 06 16:45:47 eooubuserver nextcloud.apache[347331]:                       >
Oct 06 16:45:47 eooubuserver nextcloud.apache[347331]:   Command "app:disable>
Oct 06 16:45:47 eooubuserver nextcloud.apache[347331]:                       >
Oct 06 16:45:47 eooubuserver nextcloud.apache[347331]:   Did you mean this?
Oct 06 16:45:47 eooubuserver nextcloud.apache[347331]:       app:check-code
Oct 06 16:45:47 eooubuserver nextcloud.apache[347331]:                       >
lines 2165-2185/2185 (END)

Your output text messages were truncated. Can you make the terminal wider? It looks like java is not running, right? I'm not familiar with eooubuserver or Nextcloud. Maybe it's something new that elasticsearch 7.15 uses. Mine is version 7.5. The error said "Nextcloud is not installed".

Might be easier to post the last lines of the Elasticsearch log if possible, cannot see enough of the error message shared.

Have you also tried starting Elasticsearch from just the command line and not systemd

The tail lines of Elasticsearch log file :

eugene@eooubuserver:~$ sudo cat /var/log/Elasticsearch/Elasticsearch.log | tail
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
at org.Elasticsearch.node.Node.(Node.java:626) ~[Elasticsearch-7.15.0.jar:7.15.0]
at org.Elasticsearch.node.Node.(Node.java:288) ~[Elasticsearch-7.15.0.jar:7.15.0]
at org.Elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:219) ~[Elasticsearch-7.15.0.jar:7.15.0]
at org.Elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[Elasticsearch-7.15.0.jar:7.15.0]
at org.Elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399) ~[Elasticsearch-7.15.0.jar:7.15.0]
at org.Elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:167) ~[Elasticsearch-7.15.0.jar:7.15.0]
... 6 more
eugene@eooubuserver:~$


The only way I know to start elasticsearch is by the way of :
systemctl start elasticsearch.

Something you've probably already noticed is that Java stacktraces are enormous -- to figure out what is going on here, you'll need to go back further in that log, perhaps dozens or hundreds of lines depending on the exact issue. Basically, go backwards in the log (either with less or tail -n <number> with progressively larger numbers) until you find a line that includes the word Exception (with a capital E). You may need to do this multiple times to find the original cause of an error.

And, maybe a quick look for "Nextcloud" or "occ" (nextcloud command) in Elasticsearch logs:

grep -e "extcloud" -e "occ" *elastic*.log

Another thing I saw in the java error is node or node.js. That's a requirement. Check your node version.

node --version

@nahh, thanks for your suggestion. Well, below is the occurrence of the word Exception and what the log indicates is the problem with a long list, to me, of how to solve the situation I believe, but I don't know if I can do that successfully.
However, Can I completely purge the current installation and try reinstalling Elasticsearch from scratch?

Below is the part of the log file :

2021-10-07T16:30:52,178][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [e
ooubuserver] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[Failure run
ning machine learning native code. This could be due to running on an unsupported
d OS or distribution, missing OS libraries, or a problem with the temp directory
. To bypass this problem by running Elasticsearch without machine learning functionality set [xpack.ml.enabled: false].]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:171
) ~[elasticsearch-7.15.0.jar:7.15.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:
158) ~[elasticsearch-7.15.0.jar:7.15.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwar
eCommand.java:75) ~[elasticsearch-7.15.0.jar:7.15.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:1
14) ~[elasticsearch-cli-7.15.0.jar:7.15.0]
        at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-c
li-7.15.0.jar:7.15.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:123
) ~[elasticsearch-7.15.0.jar:7.15.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)
 ~[elasticsearch-7.15.0.jar:7.15.0]
Caused by: org.elasticsearch.ElasticsearchException: Failure running machine lea
:
My system is running Linux ubuntu server 20.04 on a Compaq laptop

Again thanks.