Separating heartbeat synthetics logs from the rest

[djkprojects]

Hello,

I'm setting up Synthetic Monitoring using dockerised heartbeat image and want to separate the synthetics logs from the http/icmp ones. What I've noticed is that with the latest upgrade to 8.3 (or perhaps it was there already in previous versions) there is a template and component templates called synthetics but not being used so my question is what are these templates for?

Thank you

[TiagoQueiroz]

Hi @djkprojects,

Which component templates are you talking about? Are those the ones from Elasticsearch: Create or update component template API | Elasticsearch Guide [8.3] | Elastic ?

In the output section of the configuration file, you can add some index conditions to route events to different indexes depending on their data. Configure the Elasticsearch output | Heartbeat Reference [8.3] | Elastic

[djkprojects]

Hi,

They are synthetics-mappings and synthetics-settings and the template using these is synthetics.

What are these for?

Thanks

[TiagoQueiroz]

@lucasfcosta maybe you can shed some light here?

[Andrew_Cholakian1 (Andrew Cholakian)]

The heartbeat-* mappings are for monitors configured directly with heartbeat, while the synthetics-* ones are for ones configured via the new monitor management feature (and fleet) in the Uptime app.

In 8.4.0+ browser based monitors will only write to synthetics-*, the only reason other monitors still write to heartbeat-* is to avoid a breaking change.

As we move forward we'd like most users to transition to using monitor management and pushing monitors into kibana via the elastic/synthetics CLI tool, whereupon everything will be in synthetics-*. However, these tools are in beta status, so we don't fully recommend them yet.

See Write a synthetic test | Observability Guide [8.3] | Elastic for more info on the push command.

[lucasfcosta (Lucas da Costa)]

Reading these docs, you can see that the *-settings and *-mappings templates for synthetics configure, respectively, things like the number of shards, as described here, or the document's fieldsas described here.

If you alter those, you'll be altering the configuration for the underlying backing indices for this data stream, as these templates define those indices' attributes.

[djkprojects]

Thanks however this doesn't seem to be the case when using dockerised version of heartbeat 8.3.2 for synthetics. All browser type logs are still going to heartbeat-8.3.2 data stream which uses heartbeat template (not synthetics).

Also, both components are pretty much empty:

POST /_index_template/_simulate
{
  "index_patterns": ["dummy*"],
  "composed_of": ["synthetics-settings","synthetics-mappings"]
}

gives:

{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "synthetics"
        },
        "codec": "best_compression",
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_content"
            }
          }
        }
      }
    },
    "mappings": {
      "properties": {
        "data_stream": {
          "properties": {
            "type": {
              "type": "constant_keyword",
              "value": "synthetics"
            }
          }
        }
      }
    },
    "aliases": {}
  },
  "overlapping": []
}

What am I missing here?

The reason why I'm curious about these is because we want to use synthetics template name instead of default heartbeat but worried that it will break something.

Thanks

[djkprojects]

Thanks,

Important {es} includes the following built-in component templates:

• logs-mappings
• logs-settings
• metrics-mappings
• metrics-settings
• synthetics-mapping
• synthetics-settings

{fleet-guide}/fleet-overview.html[{agent}] uses these templates to configure backing indices for its data streams. If you use {agent} and want to overwrite one of these templates, set the version for your replacement template higher than the current version.

If you don’t use {agent} and want to disable all built-in component and index templates, set stack.templates.enabled to false using the cluster update settings API.

This answers my question.

Thank you.