Heartbeat giving data to Logstash but Logstash's data not going to Elasticsearch

Karan37 · February 14, 2024, 12:52pm

this is my logstash configuration

input {
  beats {
    port => 5044
    codec => "json_lines"
  }
}
output {
  if [type] == "heartbeat" {
    elasticsearch {
      hosts => ["https://127.0.0.1:9200"]
      index => "heartbeat-8.1.1"
      user => "elastic"
      password => "mypas"
      ssl_certificate_verification => false
    }
  }
  stdout{
    codec => "rubydebug"
  }
}

data is showing on the console of logstash that is comming from heartbeat but not going to elasticsearch
this is the data showing on the logstash console

{
           "ecs" => {
        "version" => "8.0.0"
    },
       "resolve" => {
        "rtt" => {
            "us" => 516
        },
         "ip" => "127.0.0.1"
    },
          "http" => {
             "rtt" => {
                    "content" => {
                "us" => 639
            },
                      "total" => {
                "us" => 2331
            },
            "response_header" => {
                "us" => 548
            },
              "write_request" => {
                "us" => 0
            },
                   "validate" => {
                "us" => 1187
            }
        },
        "response" => {
                "headers" => {
                  "Content-Type" => "application/json; charset=utf-8",
                          "Date" => "Wed, 14 Feb 2024 12:43:22 GMT",
                "Content-Length" => "27"
            },
            "status_code" => 200,
                   "body" => {
                 "hash" => "8811a6f55cb434d10921bccf7108016db61792083bb929eef0e592e376a0db9a",
                "bytes" => 27
            },
              "mime_type" => "application/json"
        }
    },
      "@version" => "1",
         "agent" => {
                "type" => "heartbeat",
                "name" => "LAPTOP-58N3JBV9",
                  "id" => "264d665a-eda2-4004-82ea-cfd672add6df",
             "version" => "8.12.1",
        "ephemeral_id" => "e10770a7-35e1-4c4b-9a23-8c05fc1b931a"
    },
         "state" => {
        "flap_history" => [],
                "ends" => nil,
              "checks" => 27,
         "duration_ms" => "130177",
              "status" => "up",
                "down" => 0,
                  "up" => 27,
                  "id" => "default-18da7a2aa83-0",
          "started_at" => "2024-02-14T18:11:12.0673017+05:30"
    },
           "url" => {
          "path" => "/hello",
        "scheme" => "http",
          "full" => "http://localhost:8080/hello",
          "port" => 8080,
        "domain" => "localhost"
    },
    "@timestamp" => 2024-02-14T12:43:22.245Z,
       "monitor" => {
               "name" => "My Hello World",
             "status" => "up",
               "type" => "http",
           "timespan" => {
             "lt" => "2024-02-14T12:43:27.244Z",
            "gte" => "2024-02-14T12:43:22.244Z"
        },
                 "id" => "my-hello-world",
        "check_group" => "a33cfe02-cb36-11ee-b3f5-fa54f6ff3867-1",
           "duration" => {
            "us" => 2848
        },
                 "ip" => "127.0.0.1"
    },
          "tags" => [
        [0] "beats_input_raw_event"
    ],
         "event" => {
        "dataset" => "http",
           "type" => "heartbeat/summary"
    },
           "tcp" => {
        "rtt" => {
            "connect" => {
                "us" => 1144
            }
        }
    },
      "observer" => {
        "hostname" => "laptop-58n3jbv9",
              "ip" => [
            [0] "fe80::6893:4716:41e5:3e1d",
            [1] "169.254.128.205",
            [2] "fe80::a19b:b9b4:14ee:f678",
            [3] "169.254.243.52",
            [4] "fe80::ef74:83f6:7d26:111d",
            [5] "169.254.149.47",
            [6] "fe80::4a53:fbc2:c050:94a2",
            [7] "172.16.0.235"
        ],
             "mac" => [
            [0] "F8-54-F6-FF-38-67",
            [1] "FA-54-F6-FF-38-67",
            [2] "FE-54-F6-FF-38-67"
        ]
    },
       "summary" => {
              "attempt" => 1,
        "final_attempt" => true,
               "status" => "up",
         "max_attempts" => 1,
                   "up" => 1,
                 "down" => 0,
          "retry_group" => "a33cfe02-cb36-11ee-b3f5-fa54f6ff3867"
    }
}

this is my elsticsearch configuration



action.auto_create_index: '*'


# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents

xpack.security.http.ssl:
  enabled: true
  #keystore.type: PKCS12
  keystore.path: certs/http.p12
  keystore.password: essentiallyai
 # client_authentication: optional
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  #keystore.type: PKCS12 
  verification_mode: certificate
  #client_authentication: optional
  keystore.path: certs/elastic-certificates.p12
  #keystore.password: essentiallyai
  truststore.path: certs/elastic-certificates.p12
  
  #keystore.password: essentiallyai
  
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["LAPTOP-58N3JBV9"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

if anyone want additional information please feel free to ask

Rios · February 14, 2024, 8:15pm

Change in the output:
if [type] == "heartbeat" {
to:
if [agent][type] == "heartbeat" {

Karan37 · February 15, 2024, 3:56am

hey @Rios thanks for the solution
but can you tell me i don't want my heartbeat data to be in [agent][type] because after applying this my heartbeat ping is showing data but the additional information has gone
like this

can you tell me why i am not able to see the down and up count and there is also nothing in the monitors

Rios · February 15, 2024, 4:22am

Most likely you have to set http monitor params.

Karan37 · February 15, 2024, 4:44am

this is my heartbeat configuration can you point the problem

################### Heartbeat Configuration Example #########################

# This file is an example configuration file highlighting only some common options.
# The heartbeat.reference.yml file in the same directory contains all the supported options
# with detailed comments. You can use it for reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/heartbeat/index.html

############################# Heartbeat ######################################

# Define a directory from which to load monitor definitions. Definitions take the form
# of individual yaml files.
heartbeat.config.monitors:
  # Directory + glob pattern to search for configuration files
  path: ${path.config}/monitors.d/*.yml
  # If enabled, heartbeat will periodically check the config.monitors path for changes
  reload.enabled: false
  # How often to check for changes
  reload.period: 5s

# Configure monitors inline
heartbeat.monitors:
- type: http
  # Set enabled to true (or delete the following line) to enable this monitor
  enabled: true
  # ID used to uniquely identify this monitor in Elasticsearch even if the config changes
  id: my-hello-world
  # Human readable display name for this service in Uptime UI and elsewhere
  name: My Hello World
  # List of URLs to query
  urls: ["http://localhost:8080/hello"]
  # Configure task schedule
  schedule: '@every 5s'
  # Total test connection and data exchange timeout
  #timeout: 16s
  # Name of corresponding APM service, if Elastic APM is in use for the monitored service.
  #service.name: my-apm-service-name

# Experimental: Set this to true to run heartbeat monitors exactly once at startup
#heartbeat.run_once: true

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false

# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging


# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "localhost:5601"

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

Rios · February 15, 2024, 7:50am

Go to: monitors.d\sample.http.yml.disabled rename to sample.http.yml and set params.

Karan37 · February 22, 2024, 4:11am

Thanks for the reply but i did that and this is my sample.http.yml file

- type: http # monitor type `http`. Connect via HTTP an optionally verify response
  # ID used to uniquely identify this monitor in elasticsearch even if the config changes
  id: my-http-monitor

  # Human readable display name for this service in Uptime UI and elsewhere
  name: My HTTP Monitor

  # Name of corresponding APM service, if Elastic APM is in use for the monitored service.
  #service.name: my-apm-service-name

  # Enable/Disable monitor
  #enabled: true

  # Configure task schedule
  schedule: '@every 5s' # every 5 seconds from start of beat

  # Configure URLs to ping
  hosts: ["http://localhost:8080/hello"]

  # Configure IP protocol types to ping on if hostnames are configured.
  # Ping all resolvable IPs if `mode` is `all`, or only one IP if `mode` is `any`.
  ipv4: true
  ipv6: true
  mode: any

but still facing the same issue

stephenb · February 22, 2024, 4:22am

Did you run

./heartbeats setup -e

Before connecting to logstash.

You need to do that while heartbeat is pointing at elasticsearch to load all the templates. So the schem is correct... Otherwise it'll never work

Did you do that?

Your logstash conf should look like...

Rios · February 22, 2024, 5:26am

Just to add to Stephen, HB dashboard for Kibana is not included any more. Just download from GH. It's working on 8.x

system · March 21, 2024, 5:26am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.