Elasticsearch does not start after rebooting my virtual machine

rockers · May 30, 2022, 11:31pm

last day i shut down my virtual machine and next day i open my virtual machine and run Elasticsearch it is not running and it shows this error

 sudo service elasticsearch status

× elasticsearch.service - Elasticsearch

     Loaded: loaded (/etc/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)

    Drop-In: /etc/systemd/system/elasticsearch.service.d

             └─startup-timeout.conf

     Active: failed (Result: exit-code) since Mon 2022-05-30 19:09:03 EDT; 6min ago

       Docs: https://www.elastic.co

    Process: 2871 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elast>

   Main PID: 2871 (code=exited, status=1/FAILURE)

        CPU: 3.000s

May 30 19:09:03 elkmaster-Vm systemd-entrypoint[2871]:         at org.elasticsearch.xcontent.js>

May 30 19:09:03 elkmaster-Vm systemd-entrypoint[2871]:         at org.elasticsearch.common.sett>

May 30 19:09:03 elkmaster-Vm systemd-entrypoint[2871]:         at org.elasticsearch.common.sett>

May 30 19:09:03 elkmaster-Vm systemd-entrypoint[2871]:         at org.elasticsearch.common.sett>

May 30 19:09:03 elkmaster-Vm systemd-entrypoint[2871]:         at org.elasticsearch.common.sett>

May 30 19:09:03 elkmaster-Vm systemd-entrypoint[2871]:         ... 9 more

May 30 19:09:03 elkmaster-Vm systemd[1]: elasticsearch.service: Main process exited, code=exite>

my log

[2022-05-30T07:09:02,548][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [master-1] [controller/6128] [Main.cc@174] ML controller exiting

[2022-05-30T07:09:02,638][INFO ][o.e.x.m.p.NativeController] [master-1] Native controller process has stopped - no new native processes can be started

[2022-05-30T07:09:02,848][INFO ][o.e.x.w.WatcherService   ] [master-1] stopping watch service, reason [shutdown initiated]

[2022-05-30T07:09:03,035][INFO ][o.e.x.w.WatcherLifeCycleService] [master-1] watcher has stopped and shutdown

[2022-05-30T07:09:04,461][INFO ][o.e.n.Node               ] [master-1] stopped

[2022-05-30T07:09:04,463][INFO ][o.e.n.Node               ] [master-1] closing ...

[2022-05-30T07:09:04,817][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [master-1] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-17810843837948280669/geoip-databases/UU8hIcHeTvWQr9kqd6AHsA/GeoLite2-Country.mmdb]

[2022-05-30T07:09:04,820][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [master-1] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-17810843837948280669/geoip-databases/UU8hIcHeTvWQr9kqd6AHsA/GeoLite2-ASN.mmdb]

[2022-05-30T07:09:04,820][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [master-1] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-17810843837948280669/geoip-databases/UU8hIcHeTvWQr9kqd6AHsA/GeoLite2-City.mmdb]

[2022-05-30T07:09:04,871][INFO ][o.e.n.Node               ] [master-1] closed

i used 1 master and 2 nodes, it shows status
I tried all the commands and it doesn't work
when i run this command it shows me this

sudo systemctl restart elasticsearch

Job for elasticsearch.service failed because the control process exited with error code.

See "systemctl status elasticsearch.service" and "journalctl -xeu elasticsearch.service" for details.

I checked my jvm.options and also used 512mb because i used 1gb ram so last day it works fine

run journalctl -xeu elasticsearch.service


May 30 19:47:59 elk1-Vm systemd-entrypoint[12253]:         at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:91)

May 30 19:47:59 elk1-Vm systemd-entrypoint[12253]:         at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:82)

May 30 19:47:59 elk1-Vm systemd-entrypoint[12253]:         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77)

May 30 19:47:59 elk1-Vm systemd-entrypoint[12253]:         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112)

May 30 19:47:59 elk1-Vm systemd-entrypoint[12253]:         at org.elasticsearch.cli.Command.main(Command.java:77)

May 30 19:47:59 elk1-Vm systemd-entrypoint[12253]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122)

May 30 19:47:59 elk1-Vm systemd-entrypoint[12253]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)

May 30 19:47:59 elk1-Vm systemd-entrypoint[12253]: Caused by: com.fasterxml.jackson.core.JsonParseException: Duplicate field 'cluster.name'

May 30 19:47:59 elk1-Vm systemd-entrypoint[12253]:  at [Source: (sun.nio.ch.ChannelInputStream); line: 97, column: 13]

May 30 19:47:59 elk1-Vm systemd-entrypoint[12253]:         at com.fasterxml.jackson.core.json.JsonReadContext._checkDup(JsonReadContext.java:204)

May 30 19:47:59 elk1-Vm systemd-entrypoint[12253]:         at com.fasterxml.jackson.core.json.JsonReadContext.setCurrentName(JsonReadContext.java:198)

May 30 19:47:59 elk1-Vm systemd-entrypoint[12253]:         at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:399)

May 30 19:47:59 elk1-Vm systemd-entrypoint[12253]:         at org.elasticsearch.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:73)

May 30 19:47:59 elk1-Vm systemd-entrypoint[12253]:         at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:706)

May 30 19:47:59 elk1-Vm systemd-entrypoint[12253]:         at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:675)

May 30 19:47:59 elk1-Vm systemd-entrypoint[12253]:         at org.elasticsearch.common.settings.Settings.access$400(Settings.java:75)

May 30 19:47:59 elk1-Vm systemd-entrypoint[12253]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1185)

May 30 19:47:59 elk1-Vm systemd-entrypoint[12253]:         ... 9 more

May 30 19:47:59 elk1-Vm systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE

░░ Subject: Unit process exited

░░ Defined-By: systemd

░░ Support: http://www.ubuntu.com/support

░░ 

░░ An ExecStart= process belonging to unit elasticsearch.service has exited.

░░ 

░░ The process' exit code is 'exited' and its exit status is 1.

May 30 19:47:59 elk1-Vm systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

░░ Subject: Unit failed

░░ Defined-By: systemd

░░ Support: http://www.ubuntu.com/support

░░ 

░░ The unit elasticsearch.service has entered the 'failed' state with result 'exit-code'.

May 30 19:47:59 elk1-Vm systemd[1]: Failed to start Elasticsearch.

░░ Subject: A start job for unit elasticsearch.service has failed

░░ Defined-By: systemd

░░ Support: http://www.ubuntu.com/support

░░ 

░░ A start job for unit elasticsearch.service has finished with a failure.

░░ 

░░ The job identifier is 5152 and the job result is failed.

May 30 19:47:59 elk1-Vm systemd[1]: elasticsearch.service: Consumed 3.762s CPU time.

░░ Subject: Resources consumed by unit runtime

░░ Defined-By: systemd

░░ Support: http://www.ubuntu.com/support

░░ 

░░ The unit elasticsearch.service completed and consumed the indicated resources.

leandrojmp · May 31, 2022, 2:46am

This error means that you have a duplicated entry in your elasticsearch.yml, did you change anything?

Please share your elasticsearch.yml file.

rockers · May 31, 2022, 7:22am

@leandrojmp
i deleted and recreated but this time i get a different error

fatal: [es1]: FAILED! => {"changed": true, "cmd": "/usr/share/elasticsearch/bin/elasticsearch-certutil cert -out /tmp/elastic-certificates.p12 -pass \"\"", "delta": "0:00:04.178336", "end": "2022-05-31 03:50:31.687728", "msg": "non-zero return code", "rc": 74, "start": "2022-05-31 03:50:27.509392", "stderr": "\nERROR: Output file '/tmp/elastic-certificates.p12' already exists", "stderr_lines": ["", "ERROR: Output file '/tmp/elastic-certificates.p12' already exists"], "stdout": "This tool assists you in the generation of X.509 certificates and certificate\nsigning requests for use with SSL/TLS in the Elastic stack.\n\nThe 'cert' mode generates X.509 certificate and private keys.\n    * By default, this generates a single certificate and key for use\n       on a single instance.\n    * The '-multiple' option will prompt you to enter details for multiple\n       instances and will generate a certificate and key for each one\n    * The '-in' option allows for the certificate generation to be automated by describing\n       the details of each instance in a YAML file\n\n    * An instance is any piece of the Elastic Stack that requires an SSL certificate.\n      Depending on your configuration, Elasticsearch, Logstash, Kibana, and Beats\n      may all require a certificate and private key.\n    * The minimum required value for each instance is a name. This can simply be the\n      hostname, which will be used as the Common Name of the certificate. A full\n      distinguished name may also be used.\n    * A filename value may be required for each instance. This is necessary when the\n      name would result in an invalid file or directory name. The name provided here\n      is used as the directory name (within the zip) and the prefix for the key and\n      certificate files. The filename is required if you are prompted and the name\n      is not displayed in the prompt.\n    * IP addresses and DNS names are optional. Multiple values can be specified as a\n      comma separated string. If no IP addresses or DNS names are provided, you may\n      disable hostname verification in your SSL configuration.\n\n    * All certificates generated by this tool will be signed by a certificate authority (CA)\n      unless the --self-signed command line option is specified.\n      The tool can automatically generate a new CA for you, or you can provide your own with\n      the --ca or --ca-cert command line options.\n\nBy default the 'cert' mode produces a single PKCS#12 output file which holds:\n    * The instance certificate\n    * The private key for the instance certificate\n    * The CA certificate\n\nIf you specify any of the following options:\n    * -pem (PEM formatted output)\n    * -keep-ca-key (retain generated CA key)\n    * -multiple (generate multiple certificates)\n    * -in (generate certificates from an input file)\nthen the output will be be a zip file containing individual certificate/key files\n\nNote: Generating certificates without providing a CA certificate is deprecated.\n      A CA certificate will become mandatory in the next major release.", "stdout_lines": ["This tool assists you in the generation of X.509 certificates and certificate", "signing requests for use with SSL/TLS in the Elastic stack.", "", "The 'cert' mode generates X.509 certificate and private keys.", "    * By default, this generates a single certificate and key for use", "       on a single instance.", "    * The '-multiple' option will prompt you to enter details for multiple", "       instances and will generate a certificate and key for each one", "    * The '-in' option allows for the certificate generation to be automated by describing", "       the details of each instance in a YAML file", "", "    * An instance is any piece of the Elastic Stack that requires an SSL certificate.", "      Depending on your configuration, Elasticsearch, Logstash, Kibana, and Beats", "      may all require a certificate and private key.", "    * The minimum required value for each instance is a name. This can simply be the", "      hostname, which will be used as the Common Name of the certificate. A full", "      distinguished name may also be used.", "    * A filename value may be required for each instance. This is necessary when the", "      name would result in an invalid file or directory name. The name provided here", "      is used as the directory name (within the zip) and the prefix for the key and", "      certificate files. The filename is required if you are prompted and the name", "      is not displayed in the prompt.", "    * IP addresses and DNS names are optional. Multiple values can be specified as a", "      comma separated string. If no IP addresses or DNS names are provided, you may", "      disable hostname verification in your SSL configuration.", "", "    * All certificates generated by this tool will be signed by a certificate authority (CA)", "      unless the --self-signed command line option is specified.", "      The tool can automatically generate a new CA for you, or you can provide your own with", "      the --ca or --ca-cert command line options.", "", "By default the 'cert' mode produces a single PKCS#12 output file which holds:", "    * The instance certificate", "    * The private key for the instance certificate", "    * The CA certificate", "", "If you specify any of the following options:", "    * -pem (PEM formatted output)", "    * -keep-ca-key (retain generated CA key)", "    * -multiple (generate multiple certificates)", "    * -in (generate certificates from an input file)", "then the output will be be a zip file containing individual certificate/key files", "", "Note: Generating certificates without providing a CA certificate is deprecated.", "      A CA certificate will become mandatory in the next major release."]}

how can I resolve this error for this certificate error?
I am using a local virtual machine

node1:

cluster.name: xsoar-db
node.name: node-1
node.master: true
node.data: true
network.host: 192.100.0.1
http.port: 9200
discovery.seed_hosts: ["192.100.0.1", "192.100.0.2", "192.100.0.3"]
cluster.initial_master_nodes: ["node-1", "node-2", "node-3"]
xpack.security.enabled: false
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

node 2

cluster.name: xsoar-db
node.name: node-2
node.master: true
node.data: true
network.host: 192.100.0.2
http.port: 9200
discovery.seed_hosts: ["192.100.0.1", "192.100.0.2","192.100.0.3"]
cluster.initial_master_nodes: ["node-1", "node-2", "node-3"]
xpack.security.enabled: false
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

node 3

cluster.name: xsoar-db
node.name: node-3
node.master: true
node.data: true
network.host: 192.100.0.3
http.port: 9200
discovery.seed_hosts: ["192.100.0.1", "192.100.0.2", "192.100.0.3"]
cluster.initial_master_nodes: ["node-1", "node-2", "node-3"]
xpack.security.enabled: false
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

this image based on creation

elk

log error:

`

> [2022-05-31T03:19:30,735][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2, node-3] to bootstrap a cluster: have discovered [{node-1}{xSylYs4LQ-6R-Ir6CsdSVA}{Zkru59NDTWit2WOoE_-LyQ}{192.100.0.1}{192.0.0.1:9300}{cdfhilmrstw}]; discovery will continue using [192.168.0.210:9300, 192.100.0.3:9300] from hosts providers and [{node-1}{xSylYs4LQ-6R-Ir6CsdSVA}{Zkru59NDTWit2WOoE_-LyQ}{192.100.0.1}{192.100.0.1:9300}{cdfhilmrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0

`

Christian_Dahlqvist · May 31, 2022, 7:25am

Why do you have the same ip listed 3 times?

Can the nodes connect to port 9300 across the VMs on the specified ip addresses?

rockers · May 31, 2022, 7:38am

sorry edited my post
yes client vm machine

Christian_Dahlqvist · May 31, 2022, 7:46am

What do you mean by this?

Can you log into the 192.100.0.1 node and suggessfully telnet to 192.100.0.2:9300 and 192.100.0.3:9300?

rockers · May 31, 2022, 9:59am

actually i am using virtualbox to install this 3 VM to connect

system · June 28, 2022, 10:00am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.