ElasticSearch/Kibana install on CENT/REHL

MKirby · August 5, 2021, 6:49pm

Good Afternoon;

I have been working with ELK now for 3.5 months and as I am going to be deploying this in RHEL in the near future it was time to move away from the Windows deployment that I have been testing with. So I stood up a CENTOS-7 machine and downloaded the 7.14.0 rpms for ES, Logstash, and Kibana. I installed them via the Terminal in the following way:

rpm –ivh elasticsearch.7.14.0-x86_64.rpm.  
     This installs the ElasticSearch and prepares it to run.  To have the ElasticSearch  service run automatically using systemd run the commands below:
     a. run systemctl daemon-reload
     b. run systemctl enable elasticsearch.service

I switched out the elasticsearch for logstash and kibana as needed. I have not made any changes to the basic .yml files and when I attempt to open Kibana I receive a message that I am unable to connect.

Everything is installed on 1 system. I am using localhost:5601 and have not configured users or anything extra. Please let me know what I am missing?

stephenb · August 5, 2021, 8:28pm

Are you trying to connect to Kibana from a host with a browser other than where it is installed?

The defaults bind Kibana to localhost, (only browser on that host will work) you need to make Kibana accessible from the network.

See setting here

In your kibana.yml try setting

server.host: 0.0.0.0

MKirby · August 6, 2021, 11:43am

currently I am only trying to connect to the Kibana on the server I have it installed on.

I will look at the settings you have linked to.

Thank you, I will update you with the progress after.

MKirby · August 6, 2021, 12:58pm

I made the change in the kibana.yml file to have the server.host=0.0.0.0. Saved the changes and now I am able to get a message stating "Kibana Server Not Yet Ready"

I am looking through both my Elasticstash and Kibana yml files, comparing them to the ones on my windows install but not finding too much in the way of differences. Should I also change the network.host in the the ES.yml file to 0.0.0.0 from the standard 192.168.0.1?

When I run systemctl status elasticsearch.service, I receive a response of Active: failed. I believe this means that the elasticsearch service is not running.

Am I on the right path?

Sreekanth_Ragi · August 6, 2021, 1:08pm

Yes it means that ES service is not running, we will have to get that into Running state and ensure Kibana is able to connect to it, before accessing Kibana.

A Look at ES logs might give a hint on what is stopping ES to come up.

MKirby · August 6, 2021, 1:45pm

Here is the only logs that I can find under the /var/Log/elasticsearch/gc.log

021-08-06T13:12:50.247+0000][1249][gc] Using G1
[2021-08-06T13:12:52.811+0000][1249][gc,init] Version: 16.0.1+9 (release)
[2021-08-06T13:12:52.812+0000][1249][gc,init] CPUs: 1 total, 1 available
[2021-08-06T13:12:52.812+0000][1249][gc,init] Memory: 1837M
[2021-08-06T13:12:52.812+0000][1249][gc,init] Large Page Support: Disabled
[2021-08-06T13:12:52.812+0000][1249][gc,init] NUMA Support: Disabled
[2021-08-06T13:12:52.812+0000][1249][gc,init] Compressed Oops: Enabled (32-bit)
[2021-08-06T13:12:52.812+0000][1249][gc,init] Heap Region Size: 4M
[2021-08-06T13:12:52.812+0000][1249][gc,init] Heap Min Capacity: 920M
[2021-08-06T13:12:52.812+0000][1249][gc,init] Heap Initial Capacity: 920M
[2021-08-06T13:12:52.812+0000][1249][gc,init] Heap Max Capacity: 920M
[2021-08-06T13:12:52.812+0000][1249][gc,init] Pre-touch: Enabled
[2021-08-06T13:12:52.813+0000][1249][gc,init] Parallel Workers: 1
[2021-08-06T13:12:52.813+0000][1249][gc,init] Concurrent Workers: 1
[2021-08-06T13:12:52.813+0000][1249][gc,init] Concurrent Refinement Workers: 1
[2021-08-06T13:12:52.813+0000][1249][gc,init] Periodic GC: Disabled
[2021-08-06T13:12:52.954+0000][1249][gc,metaspace] CDS archive(s) mapped at: [0x0000000800000000-0x0000000800bea000-0x0000000800bea000), size 12492800, SharedBaseAddress: 0x0000000800000000, ArchiveRelocationMode: 0.
[2021-08-06T13:12:52.954+0000][1249][gc,metaspace] Compressed class space mapped at: 0x0000000800c00000-0x0000000840c00000, reserved size: 1073741824
[2021-08-06T13:12:52.954+0000][1249][gc,metaspace] Narrow klass base: 0x0000000800000000, Narrow klass shift: 3, Narrow klass range: 0x100000000
[2021-08-06T13:13:05.301+0000][1249][safepoint   ] Safepoint "Cleanup", Time since last: 9001986347 ns, Reaching safepoint: 80803877 ns, At safepoint: 14225 ns, Total: 80818102 ns
[2021-08-06T13:13:06.500+0000][1249][safepoint   ] Safepoint "Cleanup", Time since last: 1199108098 ns, Reaching safepoint: 606464 ns, At safepoint: 4046 ns, Total: 610510 ns
[2021-08-06T13:13:07.516+0000][1249][safepoint   ] Safepoint "Cleanup", Time since last: 1000205836 ns, Reaching safepoint: 15874337 ns, At safepoint: 10246 ns, Total: 15884583 ns
[2021-08-06T13:13:10.087+0000][1249][safepoint   ] Safepoint "Cleanup", Time since last: 2570304245 ns, Reaching safepoint: 54524 ns, At safepoint: 6537 ns, Total: 61061 ns
[2021-08-06T13:13:14.249+0000][1249][safepoint   ] Safepoint "Cleanup", Time since last: 4000710110 ns, Reaching safepoint: 161916502 ns, At safepoint: 7062 ns, Total: 161923564 ns
[2021-08-06T13:13:16.104+0000][1249][gc,heap,exit] Heap
[2021-08-06T13:13:16.344+0000][1249][gc,heap,exit]  garbage-first heap   total 942080K, used 9083K [0x00000000c6800000, 0x0000000100000000)
[2021-08-06T13:13:16.344+0000][1249][gc,heap,exit]   region size 4096K, 2 young (8192K), 0 survivors (0K)
[2021-08-06T13:13:16.344+0000][1249][gc,heap,exit]  Metaspace       used 368K, committed 512K, reserved 1056768K
[2021-08-06T13:13:16.344+0000][1249][gc,heap,exit]   class space    used 18K, committed 128K, reserved 1048576K

It is like this in all of the gc.logs

I have tried to add the 0.0.0.0 in the Discovery section of the Elasticsearch.yml file. No change

stephenb · August 6, 2021, 2:09pm

To look at logs as stated in the docs use

sudo journalctl --unit elasticsearch

I would not change the discovery settings.

Technically if you installed Kibana and Elasticsearch on the same hosts , and only access from that same host and changed no settings it should work.

MKirby · August 6, 2021, 2:57pm

Reply from the journalctl command indicates that the elasticsearch service is timing out.

starting Elasticsearch
elasticsearch.service start operation timed out.  Terminating
Failed to start Elasticsearch
Unit elasticsearch.service entered failed state.
elasticsearch.service failed

What I have for a server to test this is the following:
CentOS7
2048MB Base memory
24GB hdd

I returned the hosts back to 127.0.0.1 in the ES.yml file.

stephenb · August 6, 2021, 3:09pm

There should be more logs than that... You have 2GB on Memory and you are trying to Run Elasticsearch, Kibana and Logstash on the Same host that may be challenging.

Elasticsearch will default and require 1GB of that memory, if it is not available it will not start.

To check stop all the other apps (kibana, logstash) and try to start elasticsearch first.

I suspect you may not have enough resources...

Also I would go back to the default elasticsearch.yml as it is unclear what settings you have changed

MKirby · August 6, 2021, 5:04pm

I have returned to the original elasticsearch.yml file as Stephen suggested and also added 5GB of memory to the test box. So it is now 7GB of memory. I then restarted the host machine, reloaded the virtual machine to make sure it is using the full 7GB of memory.

Next I stopped the Kibana and logstash services when the box started. Then started the elasticsearch service. Result is that the elasticsearch still times out when we try to start it.
so for fun I used su root and then tried to start the elasticsearch.service and it started. Then started the kibana service successfully, the same with logstash.

Kibana is now starting. I will take the time to put a pause or delay start on the Kibana and logstash services.

Thank you for your help. Next steps is to load the Beats I was using in the Windows install, and attempt to connect a elastic-agent.

stephenb · August 6, 2021, 5:31pm

you need to run the sudo to run the systemctl or service command for elasticsearch see here

sudo -i service elasticsearch start
sudo -i service elasticsearch stop

sudo systemctl start elasticsearch.service
sudo systemctl stop elasticsearch.service

MKirby · August 9, 2021, 2:03pm

As instructed I was able to get the Kibana and elasticsearch services running and then Kibana open. That was great.
As with all installations there needs to be user accounts created. When I add the xpack.security.enabled: true in both the Kibana.yml and elasticsearch.yml files (at the bottom of each file) as I have done in the windows installation the Kibana then does not open. Page cannot be displayed error is displayed. If I remark the xpack.security lines out and restart the services, I can get back into the Kibana. Reviewing the Kibana.yml config file in Windows I have specified a password under the kibana user

elasitcsearch.username: "kibana_system"
elasitcsearch.password:"specified pass"

Are the lines indicated above user specified or am I missing a configuration within the Kibana user interface?

Thank you for all the help and assistance. It has been great for learning and practice.

system · September 6, 2021, 2:04pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.