Elasticsearch does not start CentOS7

vassiliy.vins · November 8, 2022, 1:13am

Hi!
Installed Elasticsearch as mentioned in Elastic Docs
Generated certificates and keys - followed this link - How to install Elasticsearch and Kibana 8.0 on Centos 7

puppet here in the log - just name of the server - do not pay attention

Elasticsearch does not start and gives error:

2022-11-07T17:53:40,154][WARN ][stderr                   ] [puppet] The system environment variables are not available to Log4j due to security restrictions: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getenv.*")The system environment variables are not available to Log4j due to security restrictions: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getenv.*")The system environment variables are not available to Log4j due to security restrictions: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getenv.*")The system environment variables are not available to Log4j due to security restrictions: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getenv.*")
[2022-11-07T17:53:40,154][WARN ][stderr                   ] [puppet] The system environment variables are not available to Log4j due to security restrictions: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getenv.*")The system environment variables are not available to Log4j due to security restrictions: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getenv.*")The system environment variables are not available to Log4j due to security restrictions: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getenv.*")
[2022-11-07T17:53:40,154][WARN ][stderr                   ] [puppet] The system environment variables are not available to Log4j due to security restrictions: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getenv.*")The system environment variables are not available to Log4j due to security restrictions: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getenv.*")
[2022-11-07T17:53:40,153][WARN ][stderr                   ] [puppet] The system environment variables are not available to Log4j due to security restrictions: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getenv.*")
[2022-11-07T17:53:40,153][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [puppet] uncaught exception in thread [process reaper (pid 7127)]
java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "modifyThread")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:485) ~[?:?]
	at java.security.AccessController.checkPermission(AccessController.java:1068) ~[?:?]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]
	at org.elasticsearch.secure_sm.SecureSM.checkThreadAccess(SecureSM.java:166) ~[?:?]
	at org.elasticsearch.secure_sm.SecureSM.checkAccess(SecureSM.java:120) ~[?:?]
	at java.lang.Thread.checkAccess(Thread.java:2360) ~[?:?]
	at java.lang.Thread.setDaemon(Thread.java:2308) ~[?:?]
	at java.lang.ProcessHandleImpl.lambda$static$0(ProcessHandleImpl.java:103) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:637) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:928) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1021) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1158) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
	at jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:186) ~[?:?]

Pls, help

stephenb · November 8, 2022, 1:37am

Hi @vassiliy.vins Welcome to the community...

It is hard for us to help when users use some other / unofficial blog instead of the official docs to install the Elastic Stack with other components etc... and are unclear why the install does not work.

Why did you chose that blog / installation method? First glance it sort of looks OK but hard to say.

Not sure why you are getting that error...

Perhaps you should just look at our docs and install?

Do you have Docker on your Desktop?

Do you have access to just a plain ole CentOS Box?

vassiliy.vins · November 8, 2022, 1:02pm

Hi Stephen!

Installation was done according to official Elasticsearch web page

The unofficial blog was used ONLY for creating and signing certificates. for Elasticsearch and Kibana because official page does not describe this part in some order

For installation was used CentOS7 installed on VMware

I can easily repeat ELK installation steps from official page one more time.

Regards,

Vassiliy

stephenb · November 8, 2022, 2:14pm

If you do the default installation following the docs all the certs are done for you.

Start Elasticsearch with security enabled

When installing Elasticsearch, security features are enabled and configured by default. When you install Elasticsearch, the following security configuration occurs automatically:

Authentication and authorization are enabled, and a password is generated for the elastic built-in superuser.
Certificates and keys for TLS are generated for the transport and HTTP layer, and TLS is enabled and configured with these keys and certificates.

vassiliy.vins · November 8, 2022, 5:20pm

the reason I was using the link provided was:
Docs says - run this command
/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
I did and got output
ERROR: [xpack.security.enrollment.enabled] must be set to true to create an enrollment token
I understand that I should go and enable this xpack in elasticsearch.yml

vassiliy.vins · November 8, 2022, 5:24pm

if I go and enable this feature and run it again I will get next error
ERROR: Unable to create an enrollment token. Elasticsearch node HTTP layer SSL configuration is not configured with a keystore

What I want to say I can't see in docs steps I need to go to get running stack. Even with very basic configuration I could start to play with

But thank you for your supporting me

vassiliy.vins · November 8, 2022, 5:27pm

Actually my question is - which options should I enable in elasticsearch,yml, kibana.yml, logstash.yml to get running ELK stack. Fromn this point I could start adding certificates and play with different options

vassiliy.vins · November 8, 2022, 5:29pm

to be clear - I have now completely fresh installation ELK stack 8.5 followed ELK official Docs

vassiliy.vins · November 8, 2022, 5:37pm

after new set up I have Kibana running, Elasticsearch running, logstash running.
I have in browser for port 9200

{
  "name" : "dlx-prd-dal-search-11-p",
  "cluster_name" : "my-application",
  "cluster_uuid" : "hoBqwGgpSyClek5brPRaBA",
  "version" : {
    "number" : "8.5.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "c94b4700cda13820dad5aa74fae6db185ca5c304",
    "build_date" : "2022-10-24T16:54:16.433628434Z",
    "build_snapshot" : false,
    "lucene_version" : "9.4.1",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

And I can connect Kibana on port 5601 but I see this message:

stephenb · November 8, 2022, 6:15pm

You are seeing that in Kibana?

I think You can ignore that unless you want to use Fleet / Agent your firewall is blocking the connection to the Elastic Package Registry

Kibana connects to the Elastic Package Registry at epr.elastic.co using the Elastic Package Manager,

vassiliy.vins · November 8, 2022, 6:23pm

OK, I can ignore that. Now how can I check if kibana is able to connect to elasticsearch?
As i mentione previously I was not able to create kibana token. That;'s why I put xpack.security.enabled: 'false' in elasticsearch.yml

stephenb · November 8, 2022, 6:34pm

If you set xpack.security.enabled: 'false' you are going to mess up a few things...

You can generate a new kibana token ... if you want... but it you start turning off security make sure you know what you are doing.

if you simply clean up everything including the Elasticsearch Data Directories...

Install elasticsearch .. .it will create the kibana enrollment token in the terminal during setup.

Then install Kibana ... click on the url in the terminal during kibana install, enter the enrollment token that was generated during the elasticsearch install everything will work... all the configurations happens automatically

The whole process end to end takes about 5 minutes

But if it works .. then that is fine!

vassiliy.vins · November 8, 2022, 6:48pm

where can I find enrollment token for kibana? I didn't see it during installation, that's the reason I'm trying to recreate it.

And I've already installed kibana without this token. Can I add it somehow now?

leandrojmp · November 8, 2022, 6:57pm

The enrollment token is created during Elasticsearch installation, if you didn't copy the entire text that was printed in the screen I'm not sure you can recreate the enrollment token.

I think that it would be easy to create a service account for Kibana and use this service account to authentication in Elasticsearch.

To create a service account you need to make the following request to Elasticsearch.

curl -X POST "https://your-es-host:9200/_security/service/elastic/kibana/credential/token/kibanatoken?pretty" -u elastic:PASSWORD -k

Where PASSWORD is the password set for the elastic user.

It will return a json similar to this:

{
  "created" : true,
  "token" : {
    "name" : "kibanatoken",
    "value" : "token"
  }
}

Now you need to copy the token and configure it in Kibana.

Just put the following in your kibana.yml.

elasticsearch.serviceAccountToken: token

vassiliy.vins · November 8, 2022, 6:58pm

OK, let me try

vassiliy.vins · November 8, 2022, 7:00pm

Can I use IP instead of hostname in the command?

leandrojmp · November 8, 2022, 7:01pm

It makes no difference, you just need to make a request to your Elasticsearch.

vassiliy.vins · November 9, 2022, 1:16pm

I have tried to run command using my password and IP
curl -X POST "https://your-es-host:9200/_security/service/elastic/kibana/credential/token/kibanatoken?pretty" -u elastic:PASSWORD -k

didn't work.. output " curl: (35) SSL received a record that exceeded the maximum permissible length"

leandrojmp · November 9, 2022, 1:27pm

Never saw this error.

Do you have anything in front of your Elasticsearch, something like NGINX? Also, did you enabled https for your elasticsearch?

How did you get the status for your elasticsearch, this response:

{
  "name" : "dlx-prd-dal-search-11-p",
  "cluster_name" : "my-application",
  "cluster_uuid" : "hoBqwGgpSyClek5brPRaBA",
  "version" : {
    "number" : "8.5.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "c94b4700cda13820dad5aa74fae6db185ca5c304",
    "build_date" : "2022-10-24T16:54:16.433628434Z",
    "build_snapshot" : false,
    "lucene_version" : "9.4.1",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

vassiliy.vins · November 9, 2022, 1:35pm

No, I don't have Nginx installed
the output I provided - I got using my browser http://localhost:9200/
I read somewhere in internet - it is a way to check if elasticsearch works normally after installation. So, as you can see it works normally.
in elasticsearch.yml I enabled http.port: 9200