Elasticsearch installation

eugestel · September 28, 2021, 10:50pm

Requesting for help with Elasticsearch.
I installed Elasticsearch in my ubuntu server 20.04 but unable to get it to work.

The command : systemctl status Elasticsearch.service yielded the following:

$ systemctl status elasticsearch.service 
● elasticsearch.service - Elasticsearch
     Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
     Active: failed (Result: timeout) since Tue 2021-09-28 17:20:33 EDT; 1h 5min ago
       Docs: https://www.elastic.co
    Process: 707743 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=k>
   Main PID: 707743 (code=killed, signal=TERM)

Sep 28 17:19:14 eooubuserver systemd[1]: Starting Elasticsearch...
Sep 28 17:20:30 eooubuserver systemd[1]: elasticsearch.service: start operation timed out. Terminating.
Sep 28 17:20:33 eooubuserver systemd[1]: elasticsearch.service: Failed with result 'timeout'.
Sep 28 17:20:33 eooubuserver systemd[1]: Failed to start Elasticsearch.
eugene@eooubuserver:~$



The command " journalctl -xe"  gave out the following:

$ journalctl -xe | grep Elasticsearch | tail
Sep 28 18:46:25 eooubuserver logstash[1167]: [2021-09-28T22:46:25,237][WARN ][logstash.outputs.Elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://127.0.0.1:9200/", :exception=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
Sep 28 18:46:30 eooubuserver logstash[1167]: [2021-09-28T22:46:30,267][WARN ][logstash.outputs.Elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://127.0.0.1:9200/", :exception=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
Sep 28 18:46:35 eooubuserver logstash[1167]: [2021-09-28T22:46:35,324][WARN ][logstash.outputs.Elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://127.0.0.1:9200/", :exception=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
Sep 28 18:46:40 eooubuserver logstash[1167]: [2021-09-28T22:46:40,343][WARN ][logstash.outputs.Elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://127.0.0.1:9200/", :exception=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
Sep 28 18:46:45 eooubuserver logstash[1167]: [2021-09-28T22:46:45,383][WARN ][logstash.outputs.Elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://127.0.0.1:9200/", :exception=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
Sep 28 18:46:50 eooubuserver logstash[1167]: [2021-09-28T22:46:50,404][WARN ][logstash.outputs.Elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://127.0.0.1:9200/", :exception=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
Sep 28 18:46:55 eooubuserver logstash[1167]: [2021-09-28T22:46:55,427][WARN ][logstash.outputs.Elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://127.0.0.1:9200/", :exception=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
Sep 28 18:47:00 eooubuserver logstash[1167]: [2021-09-28T22:47:00,454][WARN ][logstash.outputs.Elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://127.0.0.1:9200/", :exception=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
Sep 28 18:47:05 eooubuserver logstash[1167]: [2021-09-28T22:47:05,490][WARN ][logstash.outputs.Elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://127.0.0.1:9200/", :exception=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
Sep 28 18:47:10 eooubuserver logstash[1167]: [2021-09-28T22:47:10,535][WARN ][logstash.outputs.Elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://127.0.0.1:9200/", :exception=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
eugene@eooubuserver:~$

Can I please get someone to help me solve this problem. Thanks.

warkolm · September 28, 2021, 11:02pm

Welcome to our community!
Please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you

What does your actual Elasticsearch logs show?

eugestel · September 28, 2021, 11:14pm

I have the following:

 sudo cat  /var/log/elasticsearch/gc.log.00 | tail
[2021-09-23T01:23:54.078+0000][3135966][gc,init] Concurrent Refinement Workers: 1
[2021-09-23T01:23:54.078+0000][3135966][gc,init] Periodic GC: Disabled
[2021-09-23T01:23:54.078+0000][3135966][gc,metaspace] CDS disabled.
[2021-09-23T01:23:54.078+0000][3135966][gc,metaspace] Compressed class space mapped at: 0x0000000100000000-0x0000000140000000, reserved size: 1073741824
[2021-09-23T01:23:54.078+0000][3135966][gc,metaspace] Narrow klass base: 0x0000000000000000, Narrow klass shift: 3, Narrow klass range: 0x140000000
[2021-09-23T01:23:55.059+0000][3135966][gc,heap,exit] Heap
[2021-09-23T01:23:55.059+0000][3135966][gc,heap,exit]  garbage-first heap   total 1048576K, used 1538K [0x00000000c0000000, 0x0000000100000000)
[2021-09-23T01:23:55.059+0000][3135966][gc,heap,exit]   region size 1024K, 2 young (2048K), 0 survivors (0K)
[2021-09-23T01:23:55.059+0000][3135966][gc,heap,exit]  Metaspace       used 3470K, committed 3520K, reserved 1056768K
[2021-09-23T01:23:55.059+0000][3135966][gc,heap,exit]   class space    used 270K, committed 320K, reserved 1048576K
eugene@eooubuserver:~$

eugestel · September 28, 2021, 11:19pm

I also have the following from syslog.:

$ cat /var/log/syslog | tail
Sep 28 19:16:19 eooubuserver nextcloud.apache[1169416]:                                                                    
Sep 28 19:16:20 eooubuserver logstash[1167]: [2021-09-28T23:16:20,165][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://127.0.0.1:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
Sep 28 19:16:20 eooubuserver nextcloud.apache[1169486]: Nextcloud is not installed - only a limited number of commands are available
Sep 28 19:16:20 eooubuserver nextcloud.apache[1169486]:                                                                    
Sep 28 19:16:20 eooubuserver nextcloud.apache[1169486]:   There are no commands defined in the "config:system" namespace.
Sep 28 19:16:20 eooubuserver nextcloud.apache[1169486]:                                                                    
Sep 28 19:16:20 eoohybrid dnf[50163]: Fedora Modular 34 - x86_64 - Updates             24 kB/s |  11 kB     00:00
Sep 28 19:16:20 eoohybrid dnf[50163]: Fedora Modular 34 - x86_64 - Updates             24 kB/s |  11 kB     00:00
Sep 28 19:16:21 eoohybrid dnf[50163]: google-chrome                                   3.7 kB/s | 1.3 kB     00:00
Sep 28 19:16:21 eoohybrid dnf[50163]: google-chrome                                   3.7 kB/s | 1.3 kB     00:00
eugene@eooubuserver:~$

warkolm · September 28, 2021, 11:22pm

You need to look in /var/log/elasticsearch/elasticsearch.log.

eugestel · September 28, 2021, 11:34pm

This is all I have in the folder /var/log/Elasticsearch :
$ sudo ls /var/log/Elasticsearch/
[sudo] password for eugene:
gc.log gc.log.02 gc.log.05 gc.log.08 gc.log.11
gc.log.00 gc.log.03 gc.log.06 gc.log.09 gc.log.12
gc.log.01 gc.log.04 gc.log.07 gc.log.10
eugene@eooubuserver:~$

eugestel · September 29, 2021, 6:43pm

Please guys, any more suggestions for me going forward?

thanhlam · September 29, 2021, 8:39pm

Hi,

Just a couple ideas to check:

Did you set up the file /etc/Elasticsearch/Elasticsearch.yml?
The content is something like this:

cluster.name: CLUSTER-NAME
node.name: ${HOSTNAME}
path.data: DATA-PATH
path.logs: LOG-DIR
network.host: BIND-ADDRESS
http.port: 9200

In that file, there's also a "path.pid" I think. That's for storing the Elasticsearch process ID when it starts. The reason for that is the following line in your systemctl status elasticsearch command output:

... -p ${PID_DIR}/elasticsearch.pid --quiet (code=k>

Try to see if you can access ${PID_DIR}.

eugestel · September 30, 2021, 12:48am

Thank you those suggestions. After effecting those, the following outputs were recorded:

$ sudo systemctl status elasticsearch
[sudo] password for eugene: 
● elasticsearch.service - Elasticsearch
     Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled;>
    Drop-In: /etc/systemd/system/elasticsearch.service.d
             └─startup-timeout.conf
     Active: activating (start) since Wed 2021-09-29 20:34:19 EDT; 2min 1s a>
       Docs: https://www.elastic.co
   Main PID: 1948 (java)
      Tasks: 19 (limit: 3396)

and

$ curl -X GET "localhost:9200/"
curl: (7) Failed to connect to localhost port 9200: Connection refused

`$ sudo cat /var/log/Elasticsearch/Elasticsearch.log | tail
[2021-09-30T00:38:34,801][INFO ][o.e.p.PluginsService ] [eooubuserver] loaded module [x-pack-shutdown]
[2021-09-30T00:38:34,802][INFO ][o.e.p.PluginsService ] [eooubuserver] loaded module [x-pack-sql]
[2021-09-30T00:38:34,802][INFO ][o.e.p.PluginsService ] [eooubuserver] loaded module [x-pack-stack]
[2021-09-30T00:38:34,827][INFO ][o.e.p.PluginsService ] [eooubuserver] loaded module [x-pack-text-structure]
[2021-09-30T00:38:34,827][INFO ][o.e.p.PluginsService ] [eooubuserver] loaded module [x-pack-voting-only-node]
[2021-09-30T00:38:34,847][INFO ][o.e.p.PluginsService ] [eooubuserver] loaded module [x-pack-watcher]
[2021-09-30T00:38:34,876][INFO ][o.e.p.PluginsService ] [eooubuserver] no plugins loaded
[2021-09-30T00:38:35,718][INFO ][o.e.e.NodeEnvironment ] [eooubuserver] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [182.8gb], net total_space [219gb], types [ext4]
[2021-09-30T00:38:35,719][INFO ][o.e.e.NodeEnvironment ] [eooubuserver] heap size [1.4gb], compressed ordinary object pointers [true]
[2021-09-30T00:38:36,264][INFO ][o.e.n.Node ] [eooubuserver] node name [eooubuserver], node ID [_wAxznjbSK6lyYiWwG2DgA], cluster name [Elasticsearch], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
eugene@eooubuserver:~$

`

warkolm · September 30, 2021, 12:49am

Was there more to the Elasticsearch log?

eugestel · September 30, 2021, 12:54am

Yes. Below is all that is produced:

$ sudo cat /var/log/elasticsearch/elasticsearch.log 
[2021-09-30T00:37:11,251][INFO ][o.e.n.Node               ] [eooubuserver] version[7.15.0], pid[1948], build[default/deb/79d65f6e357953a5b3cbcc5e2c7c21073d89aa29/2021-09-16T03:05:29.143308416Z], OS[Linux/5.4.0-88-generic/amd64], JVM[Eclipse Foundation/OpenJDK 64-Bit Server VM/16.0.2/16.0.2+7]
[2021-09-30T00:37:11,948][INFO ][o.e.n.Node               ] [eooubuserver] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2021-09-30T00:37:11,981][INFO ][o.e.n.Node               ] [eooubuserver] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-11001875298289442840, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms1461m, -Xmx1461m, -XX:MaxDirectMemorySize=766509056, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2021-09-30T00:38:34,408][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [aggs-matrix-stats]
[2021-09-30T00:38:34,436][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [analysis-common]
[2021-09-30T00:38:34,436][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [constant-keyword]
[2021-09-30T00:38:34,437][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [frozen-indices]
[2021-09-30T00:38:34,452][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [ingest-common]
[2021-09-30T00:38:34,453][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [ingest-geoip]
[2021-09-30T00:38:34,460][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [ingest-user-agent]
[2021-09-30T00:38:34,460][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [kibana]
[2021-09-30T00:38:34,476][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [lang-expression]
[2021-09-30T00:38:34,504][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [lang-mustache]
[2021-09-30T00:38:34,504][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [lang-painless]
[2021-09-30T00:38:34,506][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [mapper-extras]
[2021-09-30T00:38:34,521][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [mapper-version]
[2021-09-30T00:38:34,540][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [parent-join]
[2021-09-30T00:38:34,540][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [percolator]
[2021-09-30T00:38:34,541][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [rank-eval]
[2021-09-30T00:38:34,550][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [reindex]
[2021-09-30T00:38:34,551][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [repositories-metering-api]
[2021-09-30T00:38:34,551][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [repository-encrypted]
[2021-09-30T00:38:34,565][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [repository-url]
[2021-09-30T00:38:34,581][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [runtime-fields-common]
[2021-09-30T00:38:34,620][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [search-business-rules]
[2021-09-30T00:38:34,628][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [searchable-snapshots]
[2021-09-30T00:38:34,632][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [snapshot-repo-test-kit]
[2021-09-30T00:38:34,642][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [spatial]
[2021-09-30T00:38:34,642][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [systemd]
[2021-09-30T00:38:34,669][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [transform]
[2021-09-30T00:38:34,670][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [transport-netty4]
[2021-09-30T00:38:34,670][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [unsigned-long]
[2021-09-30T00:38:34,684][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [vector-tile]
[2021-09-30T00:38:34,685][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [vectors]
[2021-09-30T00:38:34,701][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [wildcard]
[2021-09-30T00:38:34,702][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-aggregate-metric]
[2021-09-30T00:38:34,716][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-analytics]
[2021-09-30T00:38:34,717][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-async]
[2021-09-30T00:38:34,718][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-async-search]
[2021-09-30T00:38:34,718][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-autoscaling]
[2021-09-30T00:38:34,728][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-ccr]
[2021-09-30T00:38:34,733][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-core]
[2021-09-30T00:38:34,733][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-data-streams]
[2021-09-30T00:38:34,751][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-deprecation]
[2021-09-30T00:38:34,756][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-enrich]
[2021-09-30T00:38:34,756][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-eql]
[2021-09-30T00:38:34,757][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-fleet]
[2021-09-30T00:38:34,757][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-graph]
[2021-09-30T00:38:34,770][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-identity-provider]
[2021-09-30T00:38:34,783][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-ilm]
[2021-09-30T00:38:34,788][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-logstash]
[2021-09-30T00:38:34,788][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-ml]
[2021-09-30T00:38:34,789][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-monitoring]
[2021-09-30T00:38:34,789][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-ql]
[2021-09-30T00:38:34,790][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-rollup]
[2021-09-30T00:38:34,801][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-security]
[2021-09-30T00:38:34,801][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-shutdown]
[2021-09-30T00:38:34,802][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-sql]
[2021-09-30T00:38:34,802][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-stack]
[2021-09-30T00:38:34,827][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-text-structure]
[2021-09-30T00:38:34,827][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-voting-only-node]
[2021-09-30T00:38:34,847][INFO ][o.e.p.PluginsService     ] [eooubuserver] loaded module [x-pack-watcher]
[2021-09-30T00:38:34,876][INFO ][o.e.p.PluginsService     ] [eooubuserver] no plugins loaded
[2021-09-30T00:38:35,718][INFO ][o.e.e.NodeEnvironment    ] [eooubuserver] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [182.8gb], net total_space [219gb], types [ext4]
[2021-09-30T00:38:35,719][INFO ][o.e.e.NodeEnvironment    ] [eooubuserver] heap size [1.4gb], compressed ordinary object pointers [true]
[2021-09-30T00:38:36,264][INFO ][o.e.n.Node               ] [eooubuserver] node name [eooubuserver], node ID [_wAxznjbSK6lyYiWwG2DgA], cluster name [elasticsearch], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
eugene@eooubuserver:~$

warkolm · September 30, 2021, 12:56am

There's nothing after that at all?
Does Elasticsearch respond to a curl on 9200 for the host?

eugestel · September 30, 2021, 1:01am

This is all I get . For the response from Elasticsearch what I get is :

$ curl -X GET "localhost:9200/"
curl: (7) Failed to connect to localhost port 9200: Connection refused

warkolm · September 30, 2021, 1:03am

Yeah that's what I expected.

How much memory on your host? Your heap is pretty small, but that shouldn't stop things from starting.

eugestel · September 30, 2021, 2:45am

This is what I have.:

~$ grep MemTotal /proc/meminfo 
MemTotal:        2993868 kB

thanhlam · September 30, 2021, 1:30pm

So it looks like elasticsearch is running. The logs you showed so far are all INFO messages. Can you look for some errors?

grep -i error /var/log/elasticsearch/*log

If it's memory related, you can check to see if the JVM running ok. In /etc/elasticsearch/ dir. there's a file for configuring java options: jvm.options. What options are set in there?
Can you ping localhost (127.0.0.1) or ssh to it? About the curl command, it's probably to use https by default. Try it with http. Or, just open http://localhost:9200 in a browser.

eugestel · September 30, 2021, 2:08pm

Yes I can pin 127.0.0.1 successfully:

$ ping -c 3 127.0.0.1
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.054 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.090 ms
64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.063 ms

--- 127.0.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2029ms
rtt min/avg/max/mdev = 0.054/0.069/0.090/0.015 ms

But I need help figuring the meaning of all these please.:


~$ sudo grep -i error /var/log/elasticsearch/elasticsearch.log
[2021-09-30T00:37:11,981][INFO ][o.e.n.Node               ] [eooubuserver] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-11001875298289442840, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms1461m, -Xmx1461m, -XX:MaxDirectMemorySize=766509056, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2021-09-30T13:20:01,403][INFO ][o.e.n.Node               ] [eooubuserver] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-2209817024945480120, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms1461m, -Xmx1461m, -XX:MaxDirectMemorySize=766509056, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]

$ sudo cat /etc/Elasticsearch/jvm.options
################################################################

JVM configuration

################################################################

WARNING: DO NOT EDIT THIS FILE. If you want to override the

JVM options in this file, or set any additional options, you

should create one or more files in the jvm.options.d

directory containing your adjustments.

See Settng JVM options | Elasticsearch Guide [7.15] | Elastic

for more information.

################################################################

IMPORTANT: JVM heap size

################################################################

The heap size is automatically configured by Elasticsearch

based on the available memory in your system and the roles

each node is configured to fulfill. If specifying heap is

required, it should be done through a file in jvm.options.d,

and the min and max should be set to the same value. For

example, to set the heap to 4 GB, create a new file in the

jvm.options.d directory containing these lines:

-Xms4g

-Xmx4g

See Heap size settings | Elasticsearch Guide [7.15] | Elastic

for more information

################################################################

Expert settings

################################################################

All settings below here are considered expert settings. Do

not adjust them unless you understand what you are doing. Do

not edit them in this file; instead, create a new file in the

jvm.options.d directory containing your adjustments.

################################################################

GC configuration

8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly

G1GC Configuration

NOTE: G1 GC is only supported on JDK version 10 or later

to use G1GC, uncomment the next two lines and update the version on the

following three lines to your version of the JDK

10-13:-XX:-UseConcMarkSweepGC

10-13:-XX:-UseCMSInitiatingOccupancyOnly

14-:-XX:+UseG1GC

JVM temporary directory

-Djava.io.tmpdir=${ES_TMPDIR}

heap dumps

generate a heap dump when an allocation from the Java heap fails; heap dumps

are created in the working directory of the JVM unless an alternative path is

specified

-XX:+HeapDumpOnOutOfMemoryError

specify an alternative path for heap dumps; ensure the directory exists and

has sufficient space

-XX:HeapDumpPath=/var/lib/Elasticsearch

specify an alternative path for JVM fatal error logs

-XX:ErrorFile=/var/log/Elasticsearch/hs_err_pid%p.log

JDK 8 GC logging

8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/Elasticsearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m

JDK 9+ GC logging

9-:-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/Elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m
eugene@eooubuserver:~$

eugestel · September 30, 2021, 2:11pm

~$ sudo grep -i error /var/log/elasticsearch/elasticsearch.log
[2021-09-30T00:37:11,981][INFO ][o.e.n.Node               ] [eooubuserver] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-11001875298289442840, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms1461m, -Xmx1461m, -XX:MaxDirectMemorySize=766509056, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2021-09-30T13:20:01,403][INFO ][o.e.n.Node               ] [eooubuserver] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-2209817024945480120, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms1461m, -Xmx1461m, -XX:MaxDirectMemorySize=766509056, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundledor paste code here

thanhlam · September 30, 2021, 7:21pm

It's better that you put the cat jvm.options output in a fixed format block:

like this

Anyway, there's no error in your Elasticsearch logs. Did you try curl with http (not https)?

curl "http://localhost:9200"

eugestel · October 1, 2021, 7:46pm

I did and I still get the 'connection refused' response. Can this have anything to do with iptable issues?