Metricbeat VM Can't Connect To Elasticsearch On Different Device

Elastic Stack Beats
21

I have matched the endpoints and credentials with the metricbeat on Windows Server 2019 config from my laptop, but the results are the same

config metricbeat.yml on Windows Server 2019

###################### Metricbeat Configuration Example #######################

# This file is an example configuration file highlighting only the most common
# options. The metricbeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/metricbeat/index.html


#ILM POLICY
setup.ilm.overwrite: true

# =========================== Modules configuration ============================

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false


# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
    host: "http://localhost:5601"
    username: "sameusernameonmylaptop"
    password: "samepasswordonmylaptop"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Metricbeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["https://192.168.20.13:9200"]

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "sameusernameonmylaptop"
  password: "samepasswordonmylaptop"
  ssl:
    enabled: true
    ca_trusted_fingerprint: "samefingerprintonmylaptop"
# ------------------------------ Logstash Output -------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~


# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================
# Metricbeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Metricbeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring:
  #enable: true
  #elasticsearch:
    #username: "elastic"
    #password: "4UAyu6Mmd9sXngjnwP9X"
# ============================== Instrumentation ===============================

# Instrumentation support for the metricbeat.
#instrumentation:
    # Set to true to enable instrumentation of metricbeat.
    #enabled: false

    # Environment in which metricbeat is running on (eg: staging, production, etc.)
    #environment: ""

    # APM Server hosts to report instrumentation results to.
    #hosts:
    #  - http://localhost:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    #secret_token:


# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

Checking the Config and Output:

PS C:\Program Files\Metricbeat> .\metricbeat test config
Config OK
PS C:\Program Files\Metricbeat> .\metricbeat test output
elasticsearch: https://192.168.20.13:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 192.168.20.13
    dial up... ERROR dial tcp 192.168.20.13:9200: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
22

No clue...

Try commenting those out and try putting in this

ssl.verification_mode: none

See here

This is a way to test SSL without the verification...

Then try to test again

It is still unclear if you have connectivity, your setup is unusual

23

Hi @stephenb

i still can't connect from Windows Server 2019 to my laptop, i don't know why, maybe you're right my setup is unusual.

24

If curl works then metricbeat should work assuming you are using the same IP and U/P ... if not that is highly unusual...

Perhaps that has something to do with it... networking via RDP

Frankly I am still not clear on your setup...

25

Hi @stephenb

maybe from a remote desktop it can't be connected to my laptop, but I have tried to connect from my laptop to windows server 2019 and it can be connected. a new problem arose, I have aligned the kibana endpoint from my laptop to windows server 2019 but still can't connect to kibana windows server 2019

metricbeat test output and setup from my laptop to windows server 2019

PS C:\Program Files\Metricbeat> .\metricbeat test output
elasticsearch: https://192.168.140.188:9200/...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 192.168.140.188
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 8.6.2
PS C:\Program Files\Metricbeat> .\metricbeat setup
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://localhost:5601/api/status fails: fail to execute the HTTP GET request: Get "http://localhost:5601/api/status": dial tcp [::1]:5601: connectex: No connection could be made because the target machine actively refused it. (status=0). Response:
26

I suspect your Kibana is not on localhost

27

on windows server 2019? still localhost

28

I don't know what you're trying to do. I can't see the configuration

Elasticsearch is on 192.168.140.188

So I thought that's where you're Kibana would be.

Put the Kibana IP in the Kibana section in metricbeat.yml

29

what I want to do is connect metricbeat from my laptop to windows server 2019 but an error occurs on kibana after I do metricbeat setup

this my configuration on metricbeat.yml from my laptop

###################### Metricbeat Configuration Example #######################

# This file is an example configuration file highlighting only the most common
# options. The metricbeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/metricbeat/index.html


#ILM POLICY
setup.ilm.overwrite: true
setup.ilm.enabled: true


# =========================== Modules configuration ============================

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false


# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
    host: "http://localhost:5601"
    username: "usernamefromwindowsserver"
    password: "passwordfromwindowsserver"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Metricbeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["https://192.168.140.188:9200/"]

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "usernamefromwindowsserver"
  password: "passwordfromwindowsserver"
  ssl:
    enabled: true
    ca_trusted_fingerprint: "fingerprintfromwindowsserver"


# ------------------------------ Logstash Output -------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~


# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================
# Metricbeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: true

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Metricbeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid: "C9EhflK7T0mKwbM8lLiX0Q"

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:
# ============================== Instrumentation ===============================

# Instrumentation support for the metricbeat.
#instrumentation:
    # Set to true to enable instrumentation of metricbeat.
    #enabled: false

    # Environment in which metricbeat is running on (eg: staging, production, etc.)
    #environment: ""

    # APM Server hosts to report instrumentation results to.
    #hosts:
    #  - http://localhost:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    #secret_token:


# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true
30 
setup.kibana:

    host: "http://localhost:5601" <!----- Use Kibana IP Here...
    username: "usernamefromwindowsserver"
    password: "passwordfromwindowsserver"
31

how do we know kibana ip? if you want to say ip 127.0.0.1 I can't connect using that ip

32

Where did you install it on what sever?

Did you install it?

It is separate install from elasticsearch.

Did you look at the Kibana docs?

33

Local Disk C on windows server

Yes

where can i find the kibana docs?

34

Google Kibana Documentation

Put in the IP of that server for Kibana config the in the metricbeat config above.

I think you can figure it out from here...

35

still same, i can't connect to kibana

PS C:\Program Files\Metricbeat> .\metricbeat setup
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://192.168.140.188:5601/api/status fails: fail to execute the HTTP GET request: Get "http://192.168.140.188:5601/api/status": dial tcp 192.168.140.188:5601: connectex: No connection could be made because the target machine actively refused it. (status=0). Response:
36

The error message is pretty straightforward ... Connection actively refused ...

Firewall, Connectivity etc... You will need to figure it out...

37

i'm trying to turn off the firewall but still can't connect to kibana from my laptop to windows server 2019 and my connectivity is no problem

38

Hi @stephenb

i tried to change server.host in kibana configuration using server.host : 0.0.0.0 but still, i still can't connect from my laptop to windows server 2019

39

Hi @stephenb

I want to provide an update, I have done some configuration and after that I tried to run .\metricbeat setup from my laptop to windows server 2019.
I checked on kibana windows server and all of a sudden i see my laptop hostname in kibana windows server 2019 but no data shows up

.\metricbeat setup that appears in the command prompt :

PS C:\Program Files\Metricbeat> .\metricbeat setup
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Exiting: error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory C:\Program Files\Metricbeat\kibana: Failed to import dashboard: Failed to load directory C:\Program Files\Metricbeat\kibana/7:
  error loading C:\Program Files\Metricbeat\kibana\7\dashboard\fac28650-7349-11e9-816b-07687310a99a.json: error dashboard asset: returned 0 to import file: fail to execute the HTTP POST request: Post "http://192.168.140.188:5605/api/saved_objects/_import?overwrite=true": dial tcp 192.168.140.188:5605: connectex: No connection could be made because the target machine actively refused it.. Response:
  error loading C:\Program Files\Metricbeat\kibana\7\dashboard\fc27a270-0b95-11ea-81bb-cf244189d349.json: error dashboard asset: returned 0 to import file: fail to execute the HTTP POST request: Post "http://192.168.140.188:5605/api/saved_objects/_import?overwrite=true": dial tcp 192.168.140.188:5605: connectex: No connection could be made because the target machine actively refused it.. Response:
  error loading C:\Program Files\Metricbeat\kibana\7\dashboard\fc5512c0-36d1-11ea-9f7a-097fe7ab3ddd.json: error dashboard asset: returned 0 to import file: fail to execute the HTTP POST request: Post "http://192.168.140.188:5605/api/saved_objects/_import?overwrite=true": dial tcp 192.168.140.188:5605: connectex: No connection could be made because the target machine actively refused it.. Response:
  error loading C:\Program Files\Metricbeat\kibana\7\dashboard\ff2fe020-32f7-11ea-a83e-25b8612d00cc.json: error dashboard asset: returned 0 to import file: fail to execute the HTTP POST request: Post "http://192.168.140.188:5605/api/saved_objects/_import?overwrite=true": dial tcp 192.168.140.188:5605: connectex: No connection could be made because the target machine actively refused it.. Response:
  error loading C:\Program Files\Metricbeat\kibana\7\dashboard\kubernetes-0a672d50-bcb1-11ec-b64f-7dd6e8e82013.json: error dashboard asset: returned 0 to import file: fail to execute the HTTP POST request: Post "http://192.168.140.188:5605/api/saved_objects/_import?overwrite=true": dial tcp 192.168.140.188:5605: connectex: No connection could be made because the target machine actively refused it.. Response:
  error loading C:\Program Files\Metricbeat\kibana\7\dashboard\kubernetes-21694370-bcb2-11ec-b64f-7dd6e8e82013.json: error dashboard asset: returned 0 to import file: fail to execute the HTTP POST request: Post "http://192.168.140.188:5605/api/saved_objects/_import?overwrite=true": dial tcp 192.168.140.188:5605: connectex: No connection could be made because the target machine actively refused it.. Response:
  error loading C:\Program Files\Metricbeat\kibana\7\dashboard\kubernetes-3912d9a0-bcb2-11ec-b64f-7dd6e8e82013.json: error dashboard asset: returned 0 to import file: fail to execute the HTTP POST request: Post "http://192.168.140.188:5605/api/saved_objects/_import?overwrite=true": dial tcp 192.168.140.188:5605: connectex: No connection could be made because the target machine actively refused it.. Response:
  error loading C:\Program Files\Metricbeat\kibana\7\dashboard\kubernetes-3d4d9290-bcb1-11ec-b64f-7dd6e8e82013.json: error dashboard asset: returned 0 to import file: fail to execute the HTTP POST request: Post "http://192.168.140.188:5605/api/saved_objects/_import?overwrite=true": dial tcp 192.168.140.188:5605: connectex: No connection could be made because the target machine actively refused it.. Response:
  error loading C:\Program Files\Metricbeat\kibana\7\dashboard\kubernetes-5be46210-bcb1-11ec-b64f-7dd6e8e82013.json: error dashboard asset: returned 0 to import file: fail to execute the HTTP POST request: Post "http://192.168.140.188:5605/api/saved_objects/_import?overwrite=true": dial tcp 192.168.140.188:5605: connectex: No connection could be made because the target machine actively refused it.. Response:
  error loading C:\Program Files\Metricbeat\kibana\7\dashboard\kubernetes-85879010-bcb1-11ec-b64f-7dd6e8e82013.json: error dashboard asset: returned 0 to import file: fail to execute the HTTP POST request: Post "http://192.168.140.188:5605/api/saved_objects/_import?overwrite=true": dial tcp 192.168.140.188:5605: connectex: No connection could be made because the target machine actively refused it.. Response:
  error loading C:\Program Files\Metricbeat\kibana\7\dashboard\kubernetes-9bf990a0-bcb1-11ec-b64f-7dd6e8e82013.json: error dashboard asset: returned 0 to import file: fail to execute the HTTP POST request: Post "http://192.168.140.188:5605/api/saved_objects/_import?overwrite=true": dial tcp 192.168.140.188:5605: connectex: No connection could be made because the target machine actively refused it.. Response:
  error loading C:\Program Files\Metricbeat\kibana\7\dashboard\kubernetes-b945b7b0-bcb1-11ec-b64f-7dd6e8e82013.json: error dashboard asset: returned 0 to import file: fail to execute the HTTP POST request: Post "http://192.168.140.188:5605/api/saved_objects/_import?overwrite=true": dial tcp 192.168.140.188:5605: connectex: No connection could be made because the target machine actively refused it.. Response:
  error loading C:\Program Files\Metricbeat\kibana\7\dashboard\kubernetes-dd081350-bcb1-11ec-b64f-7dd6e8e82013.json: error dashboard asset: returned 0 to import file: fail to execute the HTTP POST request: Post "http://192.168.140.188:5605/api/saved_objects/_import?overwrite=true": dial tcp 192.168.140.188:5605: connectex: No connection could be made because the target machine actively refused it.. Response:
  error loading C:\Program Files\Metricbeat\kibana\7\dashboard\kubernetes-e0195ce0-bcaf-11ec-b64f-7dd6e8e82013.json: error dashboard asset: returned 0 to import file: fail to execute the HTTP POST request: Post "http://192.168.140.188:5605/api/saved_objects/_import?overwrite=true": dial tcp 192.168.140.188:5605: connectex: No connection could be made because the target machine actively refused it.. Response:
  error loading C:\Program Files\Metricbeat\kibana\7\dashboard\kubernetes-ff1b3850-bcb1-11ec-b64f-7dd6e8e82013.json: error dashboard asset: returned 0 to import file: fail to execute the HTTP POST request: Post "http://192.168.140.188:5605/api/saved_objects/_import?overwrite=true": dial tcp 192.168.140.188:5605: connectex: No connection could be made because the target machine actively refused it.. Response:
  error loading C:\Program Files\Metricbeat\kibana\7\dashboard\metricbeat-vsphere-host.json: error dashboard asset: returned 0 to import file: fail to execute the HTTP POST request: Post "http://192.168.140.188:5605/api/saved_objects/_import?overwrite=true": dial tcp 192.168.140.188:5605: connectex: No connection could be made because the target machine actively refused it.. Response:
  error loading C:\Program Files\Metricbeat\kibana\7\dashboard\metricbeat-vsphere-vm.json: error dashboard asset: returned 0 to import file: fail to execute the HTTP POST request: Post "http://192.168.140.188:5605/api/saved_objects/_import?overwrite=true": dial tcp 192.168.140.188:5605: connectex: No connection could be made because the target machine actively refused it.. Response:
40

Try

.\metricbeat -e

Which will actually send data to elasticsearch ...

The errors above mean metricbeat can still not connect to Kibana to load the dashboards during setup.

It's says can't connect to port 5605 instead of 5601 which is what Kibana uses.. perhaps a typo in the Kibana section of metricbeat.yml

http://192.168.140.188:5605