Metricbeat VM Can't Connect To Elasticsearch On Different Device

Elastic Stack Beats
1

Hello i'm new on ELK, Metricbeat on my VM can't connect to my laptop for monitoring the system on VM, i already following the instruction from many source but still can't connect from Metricbeat VM to my laptop for monitoring, I trying the output in the VM to my laptop but still can't connect to the elasticsearch output on my laptop. i want monitoring system VM on my laptop but still can't connect.

OS i'm Using:

  • Laptop :
    Windows 11 Pro 64-bit
    Memory 16GB
    Version ELK:
    Elasticsearch 8.6.1
    Kibana 8.6.1
    MetricBeat 8.6.1

-VM:
Windows Server 2019
Memory 16GB
Version ELK:
Elasticsearch 8.6.2
KIbana 8.6.2
Metricbeat 8.6.2

2

Welcome to our community! :smiley:

Please share more information, things like your Metricbeat and Elasticsearch logs and config would be useful.

3

Config on VM :
metricbeat.yml :
i already change the output to elasticsearch on my laptop

###################### Metricbeat Configuration Example #######################

# This file is an example configuration file highlighting only the most common
# options. The metricbeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/metricbeat/index.html


#ILM POLICY
setup.ilm.overwrite: true

# =========================== Modules configuration ============================

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false


# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
    host: "http://localhost:5601"
    username: "myusername"
    password: "mypassword"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Metricbeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["https://myip:9200/"]

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "username_elastic_onlaptop"
  password: "password_elastic_onlaptop"
  ssl:
    enabled: true
    ca_trusted_fingerprint: "fingerprint_elastic_onlaptop"
# ------------------------------ Logstash Output -------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~


# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================
# Metricbeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Metricbeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring:
  #enable: true
  #elasticsearch:
    #username: "username_elastic_onlaptop"
    #password: "password_elastic_onlaptop"
# ============================== Instrumentation ===============================

# Instrumentation support for the metricbeat.
#instrumentation:
    # Set to true to enable instrumentation of metricbeat.
    #enabled: false

    # Environment in which metricbeat is running on (eg: staging, production, etc.)
    #environment: ""

    # APM Server hosts to report instrumentation results to.
    #hosts:
    #  - http://localhost:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    #secret_token:


# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

elasticsearch.yml :

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 0.0.0.0
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# --------------------------------- Readiness ----------------------------------
#
# Enable an unauthenticated TCP readiness endpoint on localhost
#
#readiness.port: 9399
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically      
# generated to configure Elasticsearch security features on 27-02-2023 02:40:39
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["TESTSERVER"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

after i using elasticsearch output from my laptop, and i try to test output command from metricbeat, i got this error

image
image964×180 55 KB

4

You haven't uncommented that, so it'll only be listening on localhost and nothing else.

Also, please don't post pictures of text, logs or code. They are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them :slight_smile:

5

how about that? is it different from the one you sent?

about the picture i'm sorry, i'm new on this forum

6

still got same error after i uncommented the network.host : 0.0.0.0

7

Anyone can help me? i still can't connect

8

Hi @Tw1cUser

Have you tried to curl elasticsearch from the metricbeat host

curl -k -u elastic https://<ipofelastichost>:9200

Are you running any of this in a container/ docker

9

Hi @stephenb

i think that command belongs to linux, and i don't use linux.
i run on windows 11 pro

10

You can also use Postman

11

@stephenb

after i use that and i put my ip local host for elasticsearch
this show up :

{
    "error": {
        "root_cause": [{
            "type": "security_exception",
            "reason": "missing authentication credentials for REST request [/]",
            "header": {
                "WWW-Authenticate": ["Basic realm=\"security\" charset=\"UTF-8\"", "Bearer realm=\"security\"", "ApiKey"]
            }
        }],
        "type": "security_exception",
        "reason": "missing authentication credentials for REST request [/]",
        "header": {
            "WWW-Authenticate": ["Basic realm=\"security\" charset=\"UTF-8\"", "Bearer realm=\"security\"", "ApiKey"]
        }
    },
    "status": 401
}

Am I really using reqbin.com correctly?

12

@stephenb

{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}

same error after i use on command prompt

13

Since you didn't show the commands, I have no idea if you are using it correctly it looks like it but I don't know.

When we're debugging, it's always helpful. If you show the exact command and the output, please show the exact command that you ran for the command line.

I would not use an online tool I would use the command line tool.

You're failing authentication

I assume you know the password for the elastic user.

Error messages are telling you that your elastic user password is not correct.

14

Hi @stephenb

I tried again using debugging using command prompt and you are right I entered the wrong password and after I tried again

Show Like This :

C:\Windows\System32>curl -k -u elastic https://192.168.20.13:9200
Enter host password for user 'elastic':
{
  "name" : "TW1CUSER",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "E7XXVjasRT2v8k_rkEse9g",
  "version" : {
    "number" : "8.6.1",
    "build_flavor" : "default",
    "build_type" : "zip",
    "build_hash" : "180c9830da956993e59e2cd70eb32b5e383ea42c",
    "build_date" : "2023-01-24T21:35:11.506992272Z",
    "build_snapshot" : false,
    "lucene_version" : "9.4.2",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
15

Anyone can help? I am very grateful if anyone can help me because this is my project for an internship

16

Did you run that curl command from inside the VM where metricbeat is running?

If you did and that result shows it should connect then the next step would be the metricbeat test command

.\metricbeat test output

Please show the command you ran and the output not just the output

17

Hi @stephenb

previously, I ran curl on my laptop because my laptop is the one I want to make monitoring the VM, and I use a local ip.

whereas for now, I'm running curl on the Windows Server 2019 using the local IP of the Windows Server 2019

C:\Windows\System32>curl -k -u elastic https://192.168.140.188:9200
Enter host password for user 'elastic':
{
  "name" : "TESTSERVER",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "C9EhflK7T0mKwbM8lLiX0Q",
  "version" : {
    "number" : "8.6.2",
    "build_flavor" : "default",
    "build_type" : "zip",
    "build_hash" : "2d58d0f136141f03239816a4e360a8d17b6d8f29",
    "build_date" : "2023-02-13T09:35:20.314882762Z",
    "build_snapshot" : false,
    "lucene_version" : "9.4.2",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

if I use a local IP from my laptop it will appear like this, even though on my laptop elasticsearch, kibana, and metricbeat are already running

C:\Windows\System32>curl -k -u elastic https://192.168.20.13:9200
Enter host password for user 'elastic':
curl: (7) Failed to connect to 192.168.20.13 port 9200: Timed out

i run .\metricbeat test output on local IP of the Windows Server 2019, show like this

PS C:\Program Files\Metricbeat> .\metricbeat test output
elasticsearch: https://192.168.140.188:9200/...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 192.168.140.188
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 8.6.2

and if i use IP from my laptop, show like this.

PS C:\Program Files\Metricbeat> .\metricbeat test output
elasticsearch: https://192.168.20.13:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 192.168.20.13
    dial up... ERROR dial tcp 192.168.20.13:9200: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

Elasticsearch, kibana, and metricbeat laptop and windows server 2019 are also running

18

Apologies I'm a bit confused...

This seems like you have a networking problem between your VM and the laptop which the curl indicates that or the IP you are using is not actually the IP for your laptop.

https://192.168.20.13:9200

What kind of laptop and have you just validated that that's the correct IP for the laptop?

This looks like a network IP issue. Nothing to do with elasticsearch components.

19

Hi @stephenb

I use Omen by HP laptop and i'm using ethernet.
for my windows server 2019, it's also on my laptop, where I use a remote desktop connection to access my windows server 2019

and if I want to implement from server to local link, will the configuration remain the same?

20

No clue with Remote desktop... This is most likely a networking issue between all your components.

curl needs to work between all your components, then setup the elastic components with the same endpoints and credentials.