Can I send heartbeat data to Elastic Security Serverless?

Elastic Security
1

Hello,

So still playing with Elastic Security Serverless at home. Considering Security starts with asset management, I was hoping to monitor my home assets with Heartbeat and send the data to Elastic Security Serverless.

But this seems to be much harder then I thought.

Is it actually possible to send heartbeat data to Elastic Security Serverless?

I'm not interested in the Uptime / Synthetics GUI, I only need the raw data.

I disabled ilm and enabled datastreams:

setup.ilm.enabled: false
setup.dsl.enabled: true

I created an API key with privileges:

{
  "heartbeat_writer": {
    "cluster": ["monitor"],
    "indices": [
      {
        "names": ["heartbeat-*"],
        "privileges": ["auto_configure", "create_doc"]
      }
    ]
  }
}

Set the host and api key:

hosts: ["https://custom-elastic-serverless-security-id.kb.eu-west-1.aws.elastic.cloud:443"]
api_key: "id:mysecureapikey"

But I keep getting a 401.

Jun 24 15:14:38 Z02-RPI5UD heartbeat[51430]: {"log.level":"debug","@timestamp":"2025-06-24T15:14:38.409+0200","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.LoggingDialer.func2","file.name":"transport/logging.go","file.line":43},"message":"Completed dialing successfully","service.name":"heartbeat","network.transport":"tcp","server.address":"custom-elastic-serverless-security-id.kb.eu-west-1.aws.elastic.cloud:443","ecs.version":"1.6.0"}
Jun 24 15:14:38 Z02-RPI5UD heartbeat[51430]: {"log.level":"debug","@timestamp":"2025-06-24T15:14:38.493+0200","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping","file.name":"eslegclient/connection.go","file.line":307},"message":"Ping request failed with: 401 Unauthorized:.....

Any help is welcome :slight_smile:

I feel like I'm missing something obvious or it might just not be possible at all...

Also tried with the cloud.id but that seems for Elastic Cloud Hosted and not Serverless.

Willem

2

Hi @willemdh

Here is the first pass API privileges... yeah they seem different than the docs
This worked for me allowed setup of the template and can write...

I am not sure if it is exact least privileges yet

{
  "heartbeat_writer": {
    "cluster": [
      "monitor",
      "read_pipeline",
      "manage_index_templates"
    ],
    "indices": [
      {
        "names": [
          "heartbeat-*"
        ],
        "privileges": [
          "view_index_metadata",
          "create_doc",
          "auto_configure"
        ],
        "allow_restricted_indices": false
      }
    ],
    "applications": [],
    "run_as": [],
    "metadata": {},
    "transient_metadata": {
      "enabled": true
    }
  }
}
1 Like
3

Thanks for your suggestion @stephenb

I just tried it, but I'm still getting a 401 for some reason.

Do you also use this host format?

hosts: ["https://custom-elastic-serverless-security-id.kb.eu-west-1.aws.elastic.cloud:443"]

(Where custom and id replaced the real values..)

Did you setup / prepare anything else?

I kind of miss the elasticsearch audit logs in the serverless option to be able to troubleshoot this.

Enabled debug in heartbeat, but apart from the "message":"Ping request failed with: 401 Unauthorized", it's not clear what is missing.

Full heartbeat.yml configuration:

logging.level: debug

setup.ilm.enabled: false
setup.dsl.enabled: true

heartbeat.config.monitors:
  path: ${path.config}/monitors.d/*.yml
  reload.enabled: false
  reload.period: 5s

heartbeat.monitors:
- type: http
  enabled: true
  id: outsideit
  name: OutsideIT
  urls: ["https://outsideit.net:443"]
  schedule: '@every 10s'

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

output.elasticsearch:
  hosts: ["https://<custom>-elastic-serverless-security-<id>.kb.eu-west-1.aws.elastic.cloud:443"]
  preset: balanced
  api_key: "id:<api-key>>"

Willem

4

You are pointing at Kibana endpoint not Elastic Endpoint

 ["https://<custom>-elastic-serverless-security-<id>.>>>>es<<<<.eu-west-1.aws.elastic.cloud:443"]

I also do not have these settings not sure that makes any difference

setup.ilm.enabled: false
setup.dsl.enabled: true
1 Like
5

You are pointing at Kibana endpoint not Elastic Endpoint

That will be the reason lol. :slight_smile:

I'm leaving for a holiday for 10 days, when I get back, I'll test with the es endpoint..

Massive thanks for the help!

1 Like
6

@stephenb I was able to get it to work with the new url and also had to change the api format to Beats.

image

Thanks!!