Metrics not working when using Logstash

Elastic Stack Logstash
1

Hello,

I'm using Metricbeats with Logstash. I figured out that when I set Metricbeat to send the output directly to Elasticsearch, I can see all the metrics information in the tab Observability -> Metrics. But when I send the that to Logstash, and this sends to Elastic, then I cannot see it in Metrics, but I'm getting the data, too, cause for both configurations I can see the data entering in Discover.

How can I visualize this data in Metrics when using Logstash between Metricbeat and Elastic?

Thanks.

2

Please take a look at this

And this

3

Hello Stephen and thank you for your help.

I've been following all the steps you said in the other post, but I had no luck... Following your steps solved a problem I was having with mapping, where some Metricbeat fields were changing the type to keyword instead of IP, but apart of this, I still have the issue and I'm not seeing anything on my metrics page.

Thanks!

4

post your metricbeat and logstash configurations...

Which Metrics Page : The Metrics Inventory or Metrics Dashboard

5

Hello, It's for both Pages.
In fact I just discovered a warning in Logstash when using systemctl status, it says "Could not index event to Elasticsearch" and it's coming from the field process.memory.pct ([process.memory.pct] cannot be changed from type [float] to [long]).
I can see too in Data Views that it's creating an error mapping conflict with host.ip, setting it into text instead of ip.

My Logstash configuration is like you said:

input {
  beats {
    port => 5044
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => ["localhost:9200"]
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      pipeline => "%{[@metadata][pipeline]}" 
      #user => "elastic"
      #password => "secret"
    }
  } else {
    elasticsearch {
      hosts => ["localhost:9200"]
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      #user => "elastic"
      #password => "secret"
    }
  }
}

It's important to say I commented user and password because I disabled login for the moment. This environment stills in testing, so for comfort I'm not gonna set it until prod.

And my Metricbeat YML:

###################### Metricbeat Configuration Example #######################

# This file is an example configuration file highlighting only the most common
# options. The metricbeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/metricbeat/index.html

# =========================== Modules configuration ============================

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false


# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
#setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Metricbeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~


# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================
# Metricbeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Metricbeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:

# ============================== Instrumentation ===============================

# Instrumentation support for the metricbeat.
#instrumentation:
    # Set to true to enable instrumentation of metricbeat.
    #enabled: false

    # Environment in which metricbeat is running on (eg: staging, production, etc.)
    #environment: ""

    # APM Server hosts to report instrumentation results to.
    #hosts:
    #  - http://localhost:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    #secret_token:


# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true
6

You need to clean up everything first ... this is because some of the data was run BEFORE setup...

  1. Stop Everything...
  2. Clean up Everything (All indices)
  3. Point metricbeat directoyto elasticsearch and Kibana
  4. Run metricbeat setup -e
  5. Then run metricbeat still pointing directly at elasticsearch ... let it run .. check everything
    4a. Stop Everything
  6. Then introduce logstash
  7. That is my recommendation
7

Ok. I will try and let you know. But, let me ask you something, is it necessary to create a pipeline? Or are they created automatically?

8

Pipelines are created by running (They are packaged as part of the modules)

./metricbeat setup -e

Some modules use them some do not... the system module does not but it is always best practice to run setup as that could change.

1 Like
9

Hello,
No luck... I tried reinstalling and still not working. It works till the moment I connect it to Logstash, then I can start seeing the mapping error in Data Views, and everything stops failing.

10

Ahhhhhhhhhh....You are using data streams.... apologies I should have recognized that...

Need to update the Elasticsearch output section....

Let me look... What version of everything?

1 Like
11

It's 8.1.2 for everything. Last one, I guess.

1 Like
12

@quim.gomez

Try this... it seems to work... need to think about why ... :slight_smile:

My Graphs and Inventory are working

I thought we needed to use the data_stream setting but since metricbeat set all that up we just index to the write alias...

Docs need to updated to work with streams I think...

output {

  # stdout {codec => "rubydebug"}

  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "http://localhost:9200"
      pipeline => "%{[@metadata][pipeline]}" 
      user => "elastic"
      password => "password"
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      action => "create"
      manage_template => false
    }
  } else {
    elasticsearch {
      hosts => "http://localhost:9200"
      user => "elastic"
      password => "password"
      index => "%{[@metadata][beat]}-%{[@metadata][version]}" <!-- The data stream write alias 
      action => "create" <!-- Data streams only allow create
      manage_template => false
    }
  }
}

Screen Shot 2022-04-19 at 12.41.27 PM
Screen Shot 2022-04-19 at 12.41.27 PM1920×1016 156 KB

13

Hello,

Thank you again for your help, but unfortunately I still getting the same errors... It's quite frustrating because I can't understand what's wrong with the configurations.

14

Exactly what errors... I validated that this worked.... using the method I showed you plus the new logstash output config with 8.1.2

If you have it working directly to Elasticsearch and then use the new logstash output I just gave you it should work.

Did you notice all the changes in the output section?

Did you clean up and Start over?

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.