# Heartbeat giving data to Logstash but Logstash's data not going to Elasticsearch

**URL:** <https://discuss.elastic.co/t/heartbeat-giving-data-to-logstash-but-logstashs-data-not-going-to-elasticsearch/353271>\
**Category:** Logstash\
**Created:** [February 14, 2024, 12:52pm UTC](https://discuss.elastic.co/t/heartbeat-giving-data-to-logstash-but-logstashs-data-not-going-to-elasticsearch/353271 "2024-02-14T12:52:42Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Karan37](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karan37/32/131451_2.png) [@Karan37](https://discuss.elastic.co/u/Karan37)\
**Post date:** [February 14, 2024, 12:52pm UTC](https://discuss.elastic.co/t/heartbeat-giving-data-to-logstash-but-logstashs-data-not-going-to-elasticsearch/353271/1 "2024-02-14T12:52:42Z")

</div>

this is my logstash configuration

```auto
input {
  beats {
    port => 5044
    codec => "json_lines"
  }
}
output {
  if [type] == "heartbeat" {
    elasticsearch {
      hosts => ["https://127.0.0.1:9200"]
      index => "heartbeat-8.1.1"
      user => "elastic"
      password => "mypas"
      ssl_certificate_verification => false
    }
  }
  stdout{
    codec => "rubydebug"
  }
}

```

data is showing on the console of logstash that is comming from heartbeat but not going to elasticsearch  
this is the data showing on the logstash console

```auto
{
           "ecs" => {
        "version" => "8.0.0"
    },
       "resolve" => {
        "rtt" => {
            "us" => 516
        },
         "ip" => "127.0.0.1"
    },
          "http" => {
             "rtt" => {
                    "content" => {
                "us" => 639
            },
                      "total" => {
                "us" => 2331
            },
            "response_header" => {
                "us" => 548
            },
              "write_request" => {
                "us" => 0
            },
                   "validate" => {
                "us" => 1187
            }
        },
        "response" => {
                "headers" => {
                  "Content-Type" => "application/json; charset=utf-8",
                          "Date" => "Wed, 14 Feb 2024 12:43:22 GMT",
                "Content-Length" => "27"
            },
            "status_code" => 200,
                   "body" => {
                 "hash" => "8811a6f55cb434d10921bccf7108016db61792083bb929eef0e592e376a0db9a",
                "bytes" => 27
            },
              "mime_type" => "application/json"
        }
    },
      "@version" => "1",
         "agent" => {
                "type" => "heartbeat",
                "name" => "LAPTOP-58N3JBV9",
                  "id" => "264d665a-eda2-4004-82ea-cfd672add6df",
             "version" => "8.12.1",
        "ephemeral_id" => "e10770a7-35e1-4c4b-9a23-8c05fc1b931a"
    },
         "state" => {
        "flap_history" => [],
                "ends" => nil,
              "checks" => 27,
         "duration_ms" => "130177",
              "status" => "up",
                "down" => 0,
                  "up" => 27,
                  "id" => "default-18da7a2aa83-0",
          "started_at" => "2024-02-14T18:11:12.0673017+05:30"
    },
           "url" => {
          "path" => "/hello",
        "scheme" => "http",
          "full" => "http://localhost:8080/hello",
          "port" => 8080,
        "domain" => "localhost"
    },
    "@timestamp" => 2024-02-14T12:43:22.245Z,
       "monitor" => {
               "name" => "My Hello World",
             "status" => "up",
               "type" => "http",
           "timespan" => {
             "lt" => "2024-02-14T12:43:27.244Z",
            "gte" => "2024-02-14T12:43:22.244Z"
        },
                 "id" => "my-hello-world",
        "check_group" => "a33cfe02-cb36-11ee-b3f5-fa54f6ff3867-1",
           "duration" => {
            "us" => 2848
        },
                 "ip" => "127.0.0.1"
    },
          "tags" => [
        [0] "beats_input_raw_event"
    ],
         "event" => {
        "dataset" => "http",
           "type" => "heartbeat/summary"
    },
           "tcp" => {
        "rtt" => {
            "connect" => {
                "us" => 1144
            }
        }
    },
      "observer" => {
        "hostname" => "laptop-58n3jbv9",
              "ip" => [
            [0] "fe80::6893:4716:41e5:3e1d",
            [1] "169.254.128.205",
            [2] "fe80::a19b:b9b4:14ee:f678",
            [3] "169.254.243.52",
            [4] "fe80::ef74:83f6:7d26:111d",
            [5] "169.254.149.47",
            [6] "fe80::4a53:fbc2:c050:94a2",
            [7] "172.16.0.235"
        ],
             "mac" => [
            [0] "F8-54-F6-FF-38-67",
            [1] "FA-54-F6-FF-38-67",
            [2] "FE-54-F6-FF-38-67"
        ]
    },
       "summary" => {
              "attempt" => 1,
        "final_attempt" => true,
               "status" => "up",
         "max_attempts" => 1,
                   "up" => 1,
                 "down" => 0,
          "retry_group" => "a33cfe02-cb36-11ee-b3f5-fa54f6ff3867"
    }
}

```

this is my elsticsearch configuration

```auto

action.auto_create_index: '*'

# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents

xpack.security.http.ssl:
  enabled: true
  #keystore.type: PKCS12
  keystore.path: certs/http.p12
  keystore.password: essentiallyai
 # client_authentication: optional
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  #keystore.type: PKCS12 
  verification_mode: certificate
  #client_authentication: optional
  keystore.path: certs/elastic-certificates.p12
  #keystore.password: essentiallyai
  truststore.path: certs/elastic-certificates.p12
  
  #keystore.password: essentiallyai
  
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["LAPTOP-58N3JBV9"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

```

if anyone want additional information please feel free to ask

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [February 14, 2024, 8:15pm UTC](https://discuss.elastic.co/t/heartbeat-giving-data-to-logstash-but-logstashs-data-not-going-to-elasticsearch/353271/2 "2024-02-14T20:15:17Z")

</div>

Change in the output:  
if [type] == "heartbeat" {  
to:  
` if [agent][type] == "heartbeat" {`

---

<div class="post-metadata">

**Author:** ![Karan37](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karan37/32/131451_2.png) [@Karan37](https://discuss.elastic.co/u/Karan37)\
**Post date:** [February 15, 2024, 3:56am UTC](https://discuss.elastic.co/t/heartbeat-giving-data-to-logstash-but-logstashs-data-not-going-to-elasticsearch/353271/3 "2024-02-15T03:56:08Z")

</div>

hey @Rios thanks for the solution  
but can you tell me i don't want my heartbeat data to be in [agent][type] because after applying this my heartbeat ping is showing data but the additional information has gone  
like this

 ![Screenshot 2024-02-15 092310](https://us1.discourse-cdn.com/elastic/original/3X/2/b/2b25271bfceded4c0e3a9eb3908ebd30450d2efc.png)

can you tell me why i am not able to see the down and up count and there is also nothing in the monitors

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [February 15, 2024, 4:22am UTC](https://discuss.elastic.co/t/heartbeat-giving-data-to-logstash-but-logstashs-data-not-going-to-elasticsearch/353271/4 "2024-02-15T04:22:09Z")

</div>

Most likely you have to set [http monitor](https://www.elastic.co/guide/en/beats/heartbeat/current/monitor-http-options.html) params.

---

<div class="post-metadata">

**Author:** ![Karan37](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karan37/32/131451_2.png) [@Karan37](https://discuss.elastic.co/u/Karan37)\
**Post date:** [February 15, 2024, 4:44am UTC](https://discuss.elastic.co/t/heartbeat-giving-data-to-logstash-but-logstashs-data-not-going-to-elasticsearch/353271/5 "2024-02-15T04:44:26Z")

</div>

this is my heartbeat configuration can you point the problem

```auto
################### Heartbeat Configuration Example #########################

# This file is an example configuration file highlighting only some common options.
# The heartbeat.reference.yml file in the same directory contains all the supported options
# with detailed comments. You can use it for reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/heartbeat/index.html

############################# Heartbeat ######################################

# Define a directory from which to load monitor definitions. Definitions take the form
# of individual yaml files.
heartbeat.config.monitors:
  # Directory + glob pattern to search for configuration files
  path: ${path.config}/monitors.d/*.yml
  # If enabled, heartbeat will periodically check the config.monitors path for changes
  reload.enabled: false
  # How often to check for changes
  reload.period: 5s

# Configure monitors inline
heartbeat.monitors:
- type: http
  # Set enabled to true (or delete the following line) to enable this monitor
  enabled: true
  # ID used to uniquely identify this monitor in Elasticsearch even if the config changes
  id: my-hello-world
  # Human readable display name for this service in Uptime UI and elsewhere
  name: My Hello World
  # List of URLs to query
  urls: ["http://localhost:8080/hello"]
  # Configure task schedule
  schedule: '@every 5s'
  # Total test connection and data exchange timeout
  #timeout: 16s
  # Name of corresponding APM service, if Elastic APM is in use for the monitored service.
  #service.name: my-apm-service-name

# Experimental: Set this to true to run heartbeat monitors exactly once at startup
#heartbeat.run_once: true

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false

# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "localhost:5601"

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

```

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [February 15, 2024, 7:50am UTC](https://discuss.elastic.co/t/heartbeat-giving-data-to-logstash-but-logstashs-data-not-going-to-elasticsearch/353271/6 "2024-02-15T07:50:34Z")

</div>

Go to: monitors.d\sample.http.yml.disabled rename to sample.http.yml and set params.

---

<div class="post-metadata">

**Author:** ![Karan37](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karan37/32/131451_2.png) [@Karan37](https://discuss.elastic.co/u/Karan37)\
**Post date:** [February 22, 2024, 4:11am UTC](https://discuss.elastic.co/t/heartbeat-giving-data-to-logstash-but-logstashs-data-not-going-to-elasticsearch/353271/7 "2024-02-22T04:11:13Z")

</div>

Thanks for the reply but i did that and this is my sample.http.yml file

```auto
- type: http # monitor type `http`. Connect via HTTP an optionally verify response
  # ID used to uniquely identify this monitor in elasticsearch even if the config changes
  id: my-http-monitor

  # Human readable display name for this service in Uptime UI and elsewhere
  name: My HTTP Monitor

  # Name of corresponding APM service, if Elastic APM is in use for the monitored service.
  #service.name: my-apm-service-name

  # Enable/Disable monitor
  #enabled: true

  # Configure task schedule
  schedule: '@every 5s' # every 5 seconds from start of beat

  # Configure URLs to ping
  hosts: ["http://localhost:8080/hello"]

  # Configure IP protocol types to ping on if hostnames are configured.
  # Ping all resolvable IPs if `mode` is `all`, or only one IP if `mode` is `any`.
  ipv4: true
  ipv6: true
  mode: any

```

but still facing the same issue

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [February 22, 2024, 4:22am UTC](https://discuss.elastic.co/t/heartbeat-giving-data-to-logstash-but-logstashs-data-not-going-to-elasticsearch/353271/8 "2024-02-22T04:22:33Z")

</div>

Did you run

`./heartbeats setup -e`

_ **Before** _ connecting to logstash.

You need to do that while heartbeat is pointing at elasticsearch to load all the templates. So the schem is correct... Otherwise it'll never work

Did you do that?

Your logstash conf should look like...

> **[Use ingest pipelines for parsing | Logstash Reference \[8.12\] | Elastic](https://www.elastic.co/guide/en/logstash/current/use-ingest-pipelines.html)**

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [February 22, 2024, 5:26am UTC](https://discuss.elastic.co/t/heartbeat-giving-data-to-logstash-but-logstashs-data-not-going-to-elasticsearch/353271/9 "2024-02-22T05:26:39Z")

</div>

Just to add to Stephen, HB dashboard for Kibana is not included any more. Just download from [GH](https://github.com/elastic/beats/issues/11802). It's working on 8.x 😇

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 21, 2024, 5:26am UTC](https://discuss.elastic.co/t/heartbeat-giving-data-to-logstash-but-logstashs-data-not-going-to-elasticsearch/353271/10 "2024-03-21T05:26:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.