Does Heartbeat Support Data Stream?

[Ray_Lin]

Hi Team,
I want to use data stream instead of ILM to manage the data from Heartbeat. But it seems the document didn't mention related setting about data stream.

Does Heartbeat supports data stream?

[shahzad31 (Shahzad)]

Hi @Ray_Lin Welcome to Elastic discuss forum,

Yes heartbeat does supports data stream since version 7.13.0

Here is example usage of data stream in the heartbeat.yml

Configure monitors inline

heartbeat.monitors:
- id: 4ae879a9-b5da-4132-94ba-ab6e1fcbdc6e
  name: Sample monitor 2
  revision: 1
  type: synthetics/http
  use_output: default
  meta:
    package:
    name: synthetics
    version: 0.1.28
  data_stream:
    namespace: default
  streams:
   - id: synthetics/http-http-4ae879a9-b5da-4132-94ba-ab6e1fcbdc6e
     name: Sample monitor 2
     type: http
     data_stream:
       dataset: http
       type: synthetics
     urls: 'http://elastic.co'
     service.name: APM Service Name
     schedule: '@every 5s'
     timeout: 1600
     max_redirects: 1
     proxy_url: 'http://elastic.co'
     tags:
       - tag
       - tag2

cc @andrewvc please verify above config
Please note that monitor type changes from just http to synthetics/http. same will be true for tcp,icmp and browser.

And if you use fleet integeration and add uptime monitors using UI, then in this case, heartbeat by default uses data streams.

Please let me know if this helps.

Regards

[Andrew_Cholakian1 (Andrew Cholakian)]

A few notes, I'd actually recommend against switching from http to synthetics/http they are synonyms now, but we don't officially document that method, it's intended for internal use by fleet.

Also, note that you'll need to setup the stream manually + all mappings to get the Uptime app to work right. This is currently undocumented, and not really something most users will be able to do correctly at this point.

At this juncture I would advise waiting for https://github.com/elastic/beats/pull/28450 to be merged and released. This will make using data streams without fleet much less finnicky and difficult. I would strongly advise against using them today unless you feel you can create the proper templates for the stream.

Lastly, I'll mention that the config values @shahzad31 shared are somewhat cumbersome, and the aforementioned PR should make things much much easier.