I have this error bash: ./metricbeat: cannot execute binary file: Exec format error

muradazz · April 6, 2021, 8:29pm

hello stephen
murad@murad:/var/log$ ls -l /var/log

total 832024
-rw-r--r--  1 root          root                  897 Apr  6 01:14 alternatives.log
-rw-r--r--  1 root          root                12193 Mar 28 13:38 alternatives.log.1
-rw-r--r--  1 root          root                  380 Feb 28 06:32 alternatives.log.2.gz
-rw-r--r--  1 root          root                  194 Jan 15 06:04 alternatives.log.3.gz
-rw-r--r--  1 root          root                 1702 Dec 16 20:00 alternatives.log.4.gz
-rw-r--r--  1 root          root                  154 Nov 19 06:26 alternatives.log.5.gz
-rw-r--r--  1 root          root                 6610 Oct 25 12:14 alternatives.log.6.gz
-rw-r-----  1 root          adm                     0 Mar  4 00:00 apport.log
-rw-r-----  1 root          adm                   624 Mar  3 21:52 apport.log.1
-rw-r-----  1 root          adm                   346 Feb 25 13:23 apport.log.2.gz
-rw-r-----  1 root          adm                   224 Dec  7 09:08 apport.log.3.gz
-rw-r-----  1 root          adm                   334 Nov  5 06:59 apport.log.4.gz
-rw-r-----  1 root          adm                   662 Oct 25 12:34 apport.log.5.gz
-rw-r-----  1 root          adm                   250 Oct 19 21:49 apport.log.6.gz
drwxr-xr-x  2 root          root                 4096 Apr  6 09:50 apt
-rw-r-----  1 syslog        adm              17773828 Apr  6 20:25 auth.log
-rw-r-----  1 syslog        adm              44835056 Apr  4 00:00 auth.log.1
-rw-r-----  1 syslog        adm                361622 Mar 29 00:00 auth.log.2.gz
-rw-r-----  1 syslog        adm               3185099 Mar 28 00:00 auth.log.3.gz
-rw-r-----  1 syslog        adm                488812 Mar 22 00:00 auth.log.4.gz
-rw-r--r--  1 root          root                56751 Jul 25  2018 bootstrap.log
-rw-rw----  1 root          utmp             47589888 Apr  6 20:25 btmp
-rw-rw----  1 root          utmp            208658304 Apr  1 00:00 btmp.1
-rw-r--r--  1 syslog        adm                801843 Feb 19 07:33 cloud-init.log
-rw-r--r--  1 root          root                38427 Feb 19 07:33 cloud-init-output.log
drwxr-xr-x  2 root          root                 4096 Apr  6 00:00 cups
drwxr-xr-x  3 root          root                 4096 Oct 19 19:22 dist-upgrade
-rw-r--r--  1 root          adm                 82693 Feb 19 07:33 dmesg
-rw-r--r--  1 root          adm                 82733 Dec 14 13:03 dmesg.0
-rw-r--r--  1 root          adm                 21049 Oct 25 12:18 dmesg.1.gz
-rw-r--r--  1 root          adm                 21012 Oct 25 11:55 dmesg.2.gz
-rw-r--r--  1 root          adm                 20938 Oct 22 11:36 dmesg.3.gz
-rw-r--r--  1 root          adm                 20761 Oct 20 10:47 dmesg.4.gz
-rw-r--r--  1 root          root                 9277 Apr  6 09:53 dpkg.log
-rw-r--r--  1 root          root                87720 Mar 30 06:54 dpkg.log.1
-rw-r--r--  1 root          root                16417 Feb 28 06:32 dpkg.log.2.gz
-rw-r--r--  1 root          root                 4470 Jan 30 06:54 dpkg.log.3.gz
-rw-r--r--  1 root          root                 9703 Dec 16 20:00 dpkg.log.4.gz
-rw-r--r--  1 root          root                 4991 Nov 30 17:56 dpkg.log.5.gz
-rw-r--r--  1 root          root               146715 Oct 30 06:14 dpkg.log.6.gz
drwxrwsrwx  2 elasticsearch elasticsearch        4096 Apr  6 19:31 elasticsearch
-rw-r--r--  1 root          root              2049792 Apr  4 08:43 faillog
drwx------  2 root          root                 4096 Apr  3 20:28 filebeat
-rw-r--r--  1 root          root                 3893 Mar 28 13:36 fontconfig.log
drwxr-xr-x  2 root          root                 4096 Sep  8  2020 gdm3
drwxr-xr-x  2 root          root                 4096 Oct 12 13:57 installer
drwxr-sr-x+ 3 root          systemd-journal      4096 Oct 12 14:05 journal
-rw-r-----  1 syslog        adm                113997 Apr  6 20:20 kern.log
-rw-r-----  1 syslog        adm                134848 Apr  3 23:35 kern.log.1
-rw-r-----  1 syslog        adm                  9246 Mar 28 23:21 kern.log.2.gz
-rw-r-----  1 syslog        adm                 19116 Mar 26 07:18 kern.log.3.gz
-rw-r-----  1 syslog        adm                  2029 Mar 21 23:32 kern.log.4.gz
drwxrwsrwx  2 kibana        kibana               4096 Mar 28 22:14 kibana
drwxr-xr-x  2 landscape     landscape            4096 Oct 12 14:06 landscape
-rw-rw-r--  1 root          utmp             18704352 Apr  6 17:56 lastlog
drwx------  2 root          root                 4096 Apr  6 14:58 letsencrypt
drwxr-xr-x  5 root          root                 4096 Feb 25 13:12 libvirt
drwxr-xr-x  2 root          root                 4096 Apr  5 00:00 lightdm
drwxr-x---  2 www-data      www-data             4096 Apr  4 00:00 lighttpd
drwxr-xr-x  2 logstash      root                 4096 Apr  2 17:14 logstash
drwx------  2 root          root                 4096 Apr  4 21:28 metricbeat
-rw-r-----  1 root          adm               1553828 Apr  6 20:24 monit.log
-rw-r-----  1 root          adm               1600645 Mar 31 00:00 monit.log.1
-rw-r-----  1 root          adm                 46454 Mar 24 00:00 monit.log.2.gz
-rw-r-----  1 root          adm                 63318 Mar 19 00:00 monit.log.3.gz
-rw-r-----  1 root          adm                 64098 Mar 12 00:01 monit.log.4.gz
drwxr-xr-x  2 root          adm                  4096 Apr  3 00:00 nginx
drwxr-xr-x  2 pihole        pihole               4096 Oct 24 10:29 pihole
-rw-r--r--  1 root          pihole              32635 Mar 24 20:29 pihole_debug.log
-rw-r--r--  1 pihole        pihole                  0 Apr  5 00:00 pihole-FTL.log
-rw-r--r--  1 pihole        pihole             230325 Apr  5 00:00 pihole-FTL.log.1
-rw-r--r--  1 pihole        pihole              86864 Apr  4 00:00 pihole-FTL.log.2.gz
-rw-r--r--  1 pihole        pihole              63883 Apr  3 00:00 pihole-FTL.log.3.gz
-rw-r--r--  1 pihole        pihole                  0 Apr  5 00:00 pihole.log
-rw-r--r--  1 pihole        pihole            2875685 Apr  5 00:00 pihole.log.1
-rw-r--r--  1 pihole        pihole             334207 Apr  4 00:00 pihole.log.2.gz
-rw-r--r--  1 pihole        pihole             161139 Apr  3 00:00 pihole.log.3.gz
-rw-r--r--  1 pihole        pihole             108158 Apr  2 00:00 pihole.log.4.gz
-rw-r--r--  1 pihole        pihole             129892 Apr  1 00:00 pihole.log.5.gz
-rw-r--r--  1 root          root                 1249 Apr  4 04:41 pihole_updateGravity.log
drwx------  2 root          root                 4096 Oct 19 19:24 private
-rw-r-----  1 syslog        adm               5827268 Apr  6 20:25 syslog
-rw-r-----  1 syslog        adm               7957689 Apr  6 00:00 syslog.1
-rw-r-----  1 syslog        adm                299863 Apr  5 00:00 syslog.2.gz
-rw-r-----  1 syslog        adm                491704 Apr  4 00:00 syslog.3.gz
-rw-r-----  1 syslog        adm                371759 Apr  3 00:00 syslog.4.gz
-rw-r-----  1 syslog        adm                225491 Apr  2 00:00 syslog.5.gz
-rw-r-----  1 syslog        adm                218439 Apr  1 00:00 syslog.6.gz
-rw-r-----  1 syslog        adm                543051 Mar 31 00:00 syslog.7.gz
-rw-------  1 root          root              4099584 Apr  4 08:43 tallylog
-rw-------  1 root          root                    0 Oct 19 19:33 ubuntu-advantage.log
-rw-r-----  1 syslog        adm                108676 Apr  6 20:20 ufw.log
-rw-r-----  1 syslog        adm                126715 Apr  3 23:35 ufw.log.1
-rw-r-----  1 syslog        adm                  9246 Mar 28 23:21 ufw.log.2.gz
-rw-r-----  1 syslog        adm                 18528 Mar 26 07:18 ufw.log.3.gz
-rw-r-----  1 syslog        adm                  1863 Mar 21 23:32 ufw.log.4.gz
drwxr-x---  2 root          adm                  4096 Apr  1 00:00 unattended-upgrades
-rw-r-----  1 root          adm                218988 Apr  6 20:21 vsftpd.log
-rw-r-----  1 root          adm                316860 Apr  3 23:58 vsftpd.log.1
-rw-r-----  1 root          adm                 27116 Mar 29 00:00 vsftpd.log.2
-rw-r-----  1 root          adm                136611 Mar 28 00:00 vsftpd.log.3
-rw-r-----  1 root          adm                 23425 Mar 21 23:55 vsftpd.log.4
-rw-rw-r--  1 root          utmp               137472 Apr  6 17:56 wtmp
-rw-r--r--  1 root          root                16703 Feb 19 07:33 Xorg.0.log
-rw-r--r--  1 root          root                17098 Feb 19 07:30 Xorg.0.log.old
-rw-r--r--  1 root          root                17732 Dec 10 12:36 Xorg.1.log
-rw-r--r--  1 root          root                17465 Dec 10 12:35 Xorg.1.log.old
-rw-r-----  1 xrdp          adm             502174785 Apr  6 20:25 xrdp.log
-rw-r-----  1 root          adm                135129 Mar 20 01:11 xrdp-sesman.log

and
murad@murad:/var/log$ curl localhost:9200

{
  "name" : "node-1",
  "cluster_name" : "murad",
  "cluster_uuid" : "11NpBIUMRHGE4HgUxcTZzA",
  "version" : {
    "number" : "7.12.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "78722783c38caa25a70982b5b042074cde5d3b3a",
    "build_date" : "2021-03-18T06:17:15.410153305Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"

muradazz · April 6, 2021, 8:47pm

also now i have issue with kibana

murad@murad:/var/log$ sudo -i service kibana status
● kibana.service - Kibana
     Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2021-04-06 20:14:28 UTC; 22min ago
       Docs: https://www.elastic.co
    Process: 284598 ExecStart=/usr/share/kibana/bin/kibana --logging.dest=/var/log/kibana/kibana.log --pid.file=/run/kibana/kibana>
   Main PID: 284598 (code=exited, status=1/FAILURE)

and also the journalctl -u kibana.service is

Apr 06 20:14:28 murad systemd[1]: kibana.service: Scheduled restart job, restart counter is at 3.
Apr 06 20:14:28 murad systemd[1]: Stopped Kibana.
Apr 06 20:14:28 murad systemd[1]: kibana.service: Start request repeated too quickly.
Apr 06 20:14:28 murad systemd[1]: kibana.service: Failed with result 'exit-code'.
Apr 06 20:14:28 murad systemd[1]: Failed to start Kibana.
Apr 06 20:04:24 murad kibana[283881]:  FATAL  Error: [config validation of [server].ssl]: must specify [certificate] and [key] -- >
Apr 06 20:04:24 murad systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Apr 06 20:04:24 murad systemd[1]: kibana.service: Failed with result 'exit-code'.
Apr 06 20:04:27 murad systemd[1]: kibana.service: Scheduled restart job, restart counter is at 2.
Apr 06 20:04:27 murad systemd[1]: Stopped Kibana.
Apr 06 20:04:27 murad systemd[1]: Started Kibana.
Apr 06 20:04:30 murad kibana[283906]:  FATAL  Error: [config validation of [server].ssl]: must specify [certificate] and [key] -- >
Apr 06 20:04:30 murad systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Apr 06 20:04:30 murad systemd[1]: kibana.service: Failed with result 'exit-code'.
Apr 06 20:04:33 murad systemd[1]: kibana.service: Scheduled restart job, restart counter is at 3.
Apr 06 20:04:33 murad systemd[1]: Stopped Kibana.
Apr 06 20:04:33 murad systemd[1]: kibana.service: Start request repeated too quickly.
Apr 06 20:04:33 murad systemd[1]: kibana.service: Failed with result 'exit-code'.
Apr 06 20:04:33 murad systemd[1]: Failed to start Kibana.
Apr 06 20:14:10 murad systemd[1]: Started Kibana.
Apr 06 20:14:13 murad kibana[284527]:  FATAL  Error: [config validation of [server].ssl]: must specify [certificate] and [key] -- >
Apr 06 20:14:13 murad systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Apr 06 20:14:13 murad systemd[1]: kibana.service: Failed with result 'exit-code'.
Apr 06 20:14:16 murad systemd[1]: kibana.service: Scheduled restart job, restart counter is at 1.
Apr 06 20:14:16 murad systemd[1]: Stopped Kibana.
Apr 06 20:14:16 murad systemd[1]: Started Kibana.
Apr 06 20:14:19 murad kibana[284566]:  FATAL  Error: [config validation of [server].ssl]: must specify [certificate] and [key] -- >
Apr 06 20:14:19 murad systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Apr 06 20:14:19 murad systemd[1]: kibana.service: Failed with result 'exit-code'.
Apr 06 20:14:22 murad systemd[1]: kibana.service: Scheduled restart job, restart counter is at 2.
Apr 06 20:14:22 murad systemd[1]: Stopped Kibana.
Apr 06 20:14:22 murad systemd[1]: Started Kibana.
Apr 06 20:14:24 murad kibana[284598]:  FATAL  Error: [config validation of [server].ssl]: must specify [certificate] and [key] -- >
Apr 06 20:14:24 murad systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Apr 06 20:14:24 murad systemd[1]: kibana.service: Failed with result 'exit-code'.
Apr 06 20:14:28 murad systemd[1]: kibana.service: Scheduled restart job, restart counter is at 3.
Apr 06 20:14:28 murad systemd[1]: Stopped Kibana.
Apr 06 20:14:28 murad systemd[1]: kibana.service: Start request repeated too quickly.
Apr 06 20:14:28 murad systemd[1]: kibana.service: Failed with result 'exit-code'.
Apr 06 20:14:28 murad systemd[1]: Failed to start Kibana.

the etc/kibana/kibana.yml
murad@murad:/etc/kibana$ nano kibana.yml

# Kibana is served by a back end server. This setting specifies the port to use. 
server.port: 5601
xpack.encryptedSavedObjects.encryptionKey: 'murad-is-configer-the-kibana-slerts'
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "0.0.0.0"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: ""

# The maximum payload size in bytes for incoming server requests.         
#server.maxPayloadBytes: 1048576

# The Kibana server's name.  This is used for display purposes.
server.name: "182.213.0.202"

# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://182.213.0.202:9200"]

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana_system"
#elasticsearch.password: "pass"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
server.ssl.enabled: true
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false

# Specifies the path where Kibana creates the process ID file.
#pid.file: /run/kibana/kibana.pid
  GNU nano 4.8                                                kibana.yml                                                 Modified  

# Enables you to specify a file where Kibana stores log output.
#logging.dest: stdout

# Set the value of this setting to true to suppress all logging output.
#logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
#i18n.locale: "en"

stephenb · April 6, 2021, 10:44pm

Elasticsearch is running.

Not sure what those errors are as you did not show but elasticsearch is running.

ls -l /var/log/elasticsearch/

Kibana comment out you are not running ssl. The error even says that

# server.ssl.enabled: true

[config validation of [server].ssl]: must specify [certificate] and [key]

If you take that out Kibana + Elasticsearch should work.

Also you are not going to be able Alerts because it does require security (authentication and SSL) but you do not just put that into this thread...

Before you do.... I highly... highly suggest reading at this walkthrough which wrote ..

muradazz · April 7, 2021, 6:43pm

thank you for your help ....

i had follow up the link step bu step and following the instructions but i had error in this point after insert paswords for all (elasitcsearch, kiban,apm,logstash,beats,remote momitoring )

Test with curl .

root@murad:/etc/elasticsearch/certs# curl -u "elastic:myawesomepassword" --cacert certs/selfca.pem https://localhost:9200
curl: (77) error setting certificate verify locations:
  CAfile: certs/selfca.pem
  CApath: /etc/ssl/certs
root@murad:/etc/elasticsearch/certs# ls
elastic-certificates.p12  elastic-stack-ca.p12  selfca.pem
root@murad:/etc/elasticsearch/certs#

murad@murad:/etc/elasticsearch$ sudo systemctl status kibana.service
● kibana.service - Kibana
     Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2021-04-07 18:12:29 UTC; 10min ago
       Docs: https://www.elastic.co
    Process: 370310 ExecStart=/usr/share/kibana/bin/kibana --logging.dest=/var/log/kibana/kibana.log --pid.file=/run/kibana/kibana>
   Main PID: 370310 (code=exited, status=1/FAILURE)

Apr 07 18:12:26 murad kibana[370310]:     line: 56,
Apr 07 18:12:26 murad kibana[370310]:     column: 0
Apr 07 18:12:26 murad kibana[370310]:   }
Apr 07 18:12:26 murad kibana[370310]: }
Apr 07 18:12:26 murad systemd[1]: kibana.service: Failed with result 'exit-code'.
Apr 07 18:12:29 murad systemd[1]: kibana.service: Scheduled restart job, restart counter is at 3.
Apr 07 18:12:29 murad systemd[1]: Stopped Kibana.
Apr 07 18:12:29 murad systemd[1]: kibana.service: Start request repeated too quickly.
Apr 07 18:12:29 murad systemd[1]: kibana.service: Failed with result 'exit-code'.
Apr 07 18:12:29 murad systemd[1]: Failed to start Kibana.

and the kibana.yml file is

murad@murad:/etc/kibana$ nano kibana.yml 
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601
xpack.encryptedSavedObjects.encryptionKey: 'murad-is-configer-the-kibana-slerts'
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "182.213.0.202"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: ""

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server's name.  This is used for display purposes.
server.name: "182.213.0.202"

# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://182.213.0.202:9200"]

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana_system"
#elasticsearch.password: "pass"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
server.ssl.enabled: true
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key
# server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/certs/kibana.pem
server.ssl.key: /etc/kibana/certs/kibana.key
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/selfca.pem" ]
xpack.encryptedSavedObjects.encryptionKey: "salkdjfhasldfkjhasdlfkjhasdflkasjdfhslkajfhasldkfjhasdlaksdjfh"
# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false

# Specifies the path where Kibana creates the process ID file.
#pid.file: /run/kibana/kibana.pid
  GNU nano 4.8                                                kibana.yml                                                           

# Enables you to specify a file where Kibana stores log output.
#logging.dest: stdout

# Set the value of this setting to true to suppress all logging output.
#logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
#i18n.locale: "en"

stephenb · April 7, 2021, 7:17pm

You need to be even more careful

Wrong directory

You are already I the certs directory

root@murad:/etc/elasticsearch/certs# curl -u "elastic:myawesomepassword" --cacert certs/selfca.pem https://localhost:9200
curl: (77) error setting certificate verify locations:
  CAfile: certs/selfca.pem
  CApath: /etc/ssl/certs

Either go up 1 directory or reference the cert in the current directory ( I added this to my how two)

cd /etc/elasticsearch
curl -u "elastic:myawesomepassword" --cacert certs/selfca.pem https://localhost:9200

in your kibana.yml

Looks like 2 things look like you have malformed yaml somewhere need to check that did you really leave that in?

  GNU nano 4.8                                                kibana.yml

# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["https://182.213.0.202:9200"] <!--- Should be https

muradazz · April 7, 2021, 7:54pm

thanks a lot
also in the correct directory i got error

root@murad:/etc/elasticsearch# curl -u "elastic:myawesomepassword" --cacert certs/selfca.pem https://localhost:9200
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}root@murad:/etc/elasticsearch# curl -u "elastic:myawesomepassword" --cacert certs/selfca.pem https://182.213.0.202:9200
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}root@murad:/etc/elasticsearch#

stephenb · April 7, 2021, 9:45pm

Hi @muradazz

To me it looks like either you did not setup the passwords or you are using the wrong password. That is what the message says. Did you even setup the usernames and passwords?

I can't really debug every step for you keep working at, good luck keep working at it.

Good luck!

muradazz · April 8, 2021, 3:44am

thanks a lot for your support

muradazz · April 9, 2021, 3:41pm

Hello stephen
last tow days i remove all things and install it again jump over all errors i faces above but now when i run kibana i have in the top of the browser: Kibana server is not ready yet this is i can't understand it

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: murad
#--------------------------------------xpack-----------------------------------
xpack.security.enabled: true
#xpack.security.http.ssl.supported_protocols: [ "TLSv1.2", "TLSv1.1" ]
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
# node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
# bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
# network.host: ["_eno1_", "_local_"]
network.host: 0.0.0.0
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
# http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
  GNU nano 4.8                                          elasticsearch.yml                                                    
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
discovery.type: single-node
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

ans

# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601
xpack.encryptedSavedObjects.encryptionKey: "salkdjfhasldfkjhasdlfkjhasdflkasjdfhslkajfhasldkfjhasdlaksdjfh"
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "182.213.0.202"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: ""

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server's name.  This is used for display purposes.
server.name: "murad"

# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://182.213.0.202:9200"]

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "kibana"
#elasticsearch.password: "pass"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/certs/kibana.pem
server.ssl.key: /etc/kibana/certs/kibana.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/selfca.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false

# Specifies the path where Kibana creates the process ID file.
#pid.file: /run/kibana/kibana.pid

# Enables you to specify a file where Kibana stores log output.
#logging.dest: stdout

  GNU nano 4.8                                             kibana.yml                                                        
# Set the value of this setting to true to suppress all logging output.
#logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
#i18n.locale: "en"

logstash.yml

murad@murad:/etc/logstash$ sudo nano logstash.conf 
  GNU nano 4.8                                            logstash.conf                                                      
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["https://182.213.0.202:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "elasticxxxxxxx"
    cacert => "/etc/elasticsearch/certs/elastic-certificates.p12"
    ssl => true
  }

murad@murad:/etc/logstash$

and

root@murad:/etc/elasticsearch/certs# systemctl status logstash
● logstash.service - logstash
     Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
     Active: deactivating (stop-sigterm) since Thu 2021-04-08 17:43:15 UTC; 21h ago
   Main PID: 442390 (java)
      Tasks: 75 (limit: 43307)
     Memory: 1.0G
     CGroup: /system.slice/logstash.service
             └─442390 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFract>

Apr 09 15:26:55 murad logstash[442390]: [2021-04-09T15:26:55,319][WARN ][logstash.outputs.elasticsearch][main] Attempted to >
Apr 09 15:26:58 murad logstash[442390]: [2021-04-09T15:26:58,912][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflig>
Apr 09 15:27:05 murad logstash[442390]: [2021-04-09T15:27:05,325][WARN ][logstash.outputs.elasticsearch][main] Attempted to >
Apr 09 15:27:05 murad logstash[442390]: [2021-04-09T15:27:05,325][WARN ][logstash.outputs.elasticsearch][main] Attempted to >
Apr 09 15:27:09 murad logstash[442390]: [2021-04-09T15:27:09,113][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflig>
Apr 09 15:27:10 murad logstash[442390]: [2021-04-09T15:27:10,328][WARN ][logstash.outputs.elasticsearch][main] Attempted to >
Apr 09 15:27:10 murad logstash[442390]: [2021-04-09T15:27:10,328][WARN ][logstash.outputs.elasticsearch][main] Attempted to >
lines 1-19/19 (END)

and kibana & elasticsearch servics is running good
but logstash log i have

Apr 09 15:35:10 murad logstash[442390]: [2021-04-09T15:35:10,612][WARN ][logstash.outputs.elasticsearch][main] Attempted to >
Apr 09 15:35:13 murad logstash[442390]: [2021-04-09T15:35:13,420][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflig>
Apr 09 15:35:25 murad logstash[442390]: [2021-04-09T15:35:25,621][WARN ][logstash.outputs.elasticsearch][main] Attempted to >
////
[WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>58, "na>
[WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"htt>
[WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"htt>

can you help me ?

stephenb · April 10, 2021, 2:56am

output {
  elasticsearch {
    hosts => ["https://182.213.0.202:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "elasticxxxxxxx"
    cacert => "/etc/elasticsearch/certs/elastic-certificates.p12" <-- should be selfca.pem
    ssl => true
  }

In the Logstash out I think it should be the following in needs to be that cacert (certificate authority) not the actual cert
cacert => /etc/elasticsearch/certs/selfca.pem

system · May 8, 2021, 4:57am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Introduction to Observability: Logging Elastic Training	4	533	August 29, 2020
Error initializing beat: error loading config file: stat filebeat.yml: no such file or directory Beats filebeat	4	1215	June 17, 2021
[macOS 10.15.7] Cannot execute filebeat, auditbeat, or metricbeat Beats filebeat , metricbeat , auditbeat	3	340	June 13, 2023
Filebeat 7.4.0 issue Beats filebeat	3	2579	November 6, 2019
Exec format error. Binary file not executable on FreeBSD Beats filebeat	3	3422	July 5, 2017

I have this error bash: ./metricbeat: cannot execute binary file: Exec format error

Related topics