How change index name in Heartbeat

Hi there guys
i need help with a problem i have in heartbeat.yml
look what I need is to change the name of the index of heartbeat, ie mapping to index it in kibana with another name other than heartbeat-7-10-2-2021-0000, I need to change that name, look at elasticsearch and implement the code that says, but it does not help me look at this is the code that I add in heartbeat.yml, I hope you can help me.

output.elasticsearch.index: "customname-%{[agent.version]}-%{+yyyy.MM.dd}"
setup.template.name: "customname"
setup.template.pattern: "customname-*"

and here it is in my heartbeat.yml

################### Heartbeat Configuration Example #########################

# This file is an example configuration file highlighting only some common options.
# The heartbeat.reference.yml file in the same directory contains all the supported options
# with detailed comments. You can use it for reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/heartbeat/index.html

############################# Heartbeat ######################################

# Define a directory to load monitor definitions from. Definitions take the form
# of individual yaml files.
heartbeat.config.monitors:
  # Directory + glob pattern to search for configuration files
  path: ${path.config}/monitors.d/*.yml
  # If enabled, heartbeat will periodically check the config.monitors path for changes
  reload.enabled: true
  # How often to check for changes
  reload.period: 5s

# Configure monitors inline
heartbeat.monitors:
- type: http
  # ID used to uniquely identify this monitor in elasticsearch even if the config changes
  id: my-monitor
  # Human readable display name for this service in Uptime UI and elsewhere
  name: MY ELK
  # List or urls to query
  urls: ["http://localhost:5601"]
  # Configure task schedule
  schedule: '@every 10s'
  # Total test connection and data exchange timeout
  #timeout: 16s
# Name of corresponding APM service, if Elastic APM is in use for the monitored service.
  #service_name: my-apm-service-name

# ======================= Elasticsearch template setting =======================
output.elasticsearch.index: "efecty-%{[agent.version]}-%{+yyyy.MM.dd}"
 # setup.template.enable
  setup.template.name: "efecty"
  setup.template.pattern: "efec*"
  #setup.template.overwrite: false
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #output.elasticsearch.index: "efecty-%{[agent.version]}-%{+yyyy.MM.dd}"
  #setup.template.enable:
  #setup.template.name: "efecty"
  #setup.template.pattern: "efec*"
  #_source.enabled: false

# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging


# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
 #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Heartbeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# =============================== Elastic Cloud ================================

# These settings simplify using Heartbeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  #index: "efecbeat-%{[agent.version]}-%{+yyyy.MM.dd}"
  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "elastic"
  password: "3JJcBodYI6G3tLDzT1LR"

# ------------------------------ Logstash Output -------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================

processors:
  - add_observer_metadata:
      # Optional, but recommended geo settings for the location Heartbeat is running in
      #geo:
        # Token describing this location
        #name: us-east-1a
        # Lat, Lon "
        #location: "37.926868, -78.024902"


# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================
# Heartbeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Heartbeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:

# ============================== Instrumentation ===============================

# Instrumentation support for the heartbeat.
#instrumentation:
    # Set to true to enable instrumentation of heartbeat.
    #enabled: false

    # Environment in which heartbeat is running on (eg: staging, production, etc.)
    #environment: ""

    # APM Server hosts to report instrumentation results to.
    #hosts:
    #  - http://localhost:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    #secret_token:

# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

Hello Juan,

If your cluster is a 7.x version it uses the ILM settings instead of the template settings by default: Configure index lifecycle management | Heartbeat Reference [7.11] | Elastic

This means, you have to configure the following:

setup.ilm.rollover_alias: "heartbeat"
setup.ilm.pattern: "{now/d}-000001" 

You could also disable ILM which would cause the template settings to be used but I would not recommend that:

setup.ilm.enabled: false

Best regards
Wolfram

I get this error

 heartbeat-elastic.service - Ping remote services for availability and log results to Elasticsearch or send to Logstash.
   Loaded: loaded (/lib/systemd/system/heartbeat-elastic.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2021-02-18 09:10:01 -05; 4s ago
     Docs: https://www.elastic.co/products/beats/heartbeat
  Process: 4366 ExecStart=/usr/share/heartbeat/bin/heartbeat --environment systemd $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS (code=exited, status=1/FAI
 Main PID: 4366 (code=exited, status=1/FAILURE)

feb 18 09:10:01 serverelk-VirtualBox systemd[1]: heartbeat-elastic.service: Failed with result 'exit-code'.
feb 18 09:10:01 serverelk-VirtualBox systemd[1]: heartbeat-elastic.service: Service hold-off time over, scheduling restart.
feb 18 09:10:01 serverelk-VirtualBox systemd[1]: heartbeat-elastic.service: Scheduled restart job, restart counter is at 5.
feb 18 09:10:01 serverelk-VirtualBox systemd[1]: Stopped Ping remote services for availability and log results to Elasticsearch or send to Logstash..
feb 18 09:10:01 serverelk-VirtualBox systemd[1]: heartbeat-elastic.service: Start request repeated too quickly.
feb 18 09:10:01 serverelk-VirtualBox systemd[1]: heartbeat-elastic.service: Failed with result 'exit-code'.
feb 18 09:10:01 serverelk-VirtualBox systemd[1]: Failed to start Ping remote services for availability and log results to Elasticsearch or send to Logstash..
~

and this is the heartbeat.yml

################### Heartbeat Configuration Example #########################

# This file is an example configuration file highlighting only some common options.
# The heartbeat.reference.yml file in the same directory contains all the supported options
# with detailed comments. You can use it for reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/heartbeat/index.html

############################# Heartbeat ######################################

# Define a directory to load monitor definitions from. Definitions take the form
# of individual yaml files.
heartbeat.config.monitors:
  # Directory + glob pattern to search for configuration files
  path: ${path.config}/monitors.d/*.yml
  # If enabled, heartbeat will periodically check the config.monitors path for changes
  reload.enabled: true
  # How often to check for changes
  reload.period: 5s

# Configure monitors inline
heartbeat.monitors:
- type: http
  # ID used to uniquely identify this monitor in elasticsearch even if the config changes
  id: my-monitor
  # Human readable display name for this service in Uptime UI and elsewhere
  name: MY ELK
  # List or urls to query
  urls: ["http://localhost:5601"]
  # Configure task schedule
  schedule: '@every 10s'
  # Total test connection and data exchange timeout
  #timeout: 16s
 Name of corresponding APM service, if Elastic APM is in use for the monitored service.
  #service_name: my-apm-service-name

# ======================= Elasticsearch template setting =======================
#output.elasticsearch.index: "efecty-%{[agent.version]}-%{+yyyy.MM.dd}"
 setup.ilm.rollover_alias: "efecbeat"
 setup.ilm.pattern: "{now/d}-000001"
 setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #output.elasticsearch.index: "efecty-%{[agent.version]}-%{+yyyy.MM.dd}"
  #setup.template.enable:
  #setup.template.name: "efecty"
  #setup.template.pattern: "efec*"
  #_source.enabled: false

Have you looked into the heartbeat logs? The systemd logs do not seem to contain an error..

yes, but it seems that when adding that line of code, this error occurs.

So there is nothing in the heartbeat log?

Update: have you tried enabling debug mode: Debug | Heartbeat Reference [7.11] | Elastic ?

yes, here it is

serverelk@serverelk-VirtualBox:/etc/heartbeat$ sudo heartbeat -e -c heartbeat.yml
2021-02-18T09:25:07.437-0500	INFO	instance/beat.go:645	Home path: [/usr/share/heartbeat] Config path: [/etc/heartbeat] Data path: [/var/lib/heartbeat] Logs path: [/var/log/heartbeat]
2021-02-18T09:25:07.438-0500	INFO	instance/beat.go:653	Beat ID: 57a8be40-0cef-44d7-aff3-0ee4c15294fe
2021-02-18T09:25:07.441-0500	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2021-02-18T09:25:07.442-0500	INFO	[beat]	instance/beat.go:981	Beat info	{"system_info": {"beat": {"path": {"config": "/etc/heartbeat", "data": "/var/lib/heartbeat", "home": "/usr/share/heartbeat", "logs": "/var/log/heartbeat"}, "type": "heartbeat", "uuid": "57a8be40-0cef-44d7-aff3-0ee4c15294fe"}}}
2021-02-18T09:25:07.443-0500	INFO	[beat]	instance/beat.go:990	Build info	{"system_info": {"build": {"commit": "1da173a9e716715a7a54bb3ff4db05b5c24fc8ce", "libbeat": "7.10.1", "time": "2020-12-04T23:33:19.000Z", "version": "7.10.1"}}}
2021-02-18T09:25:07.443-0500	INFO	[beat]	instance/beat.go:993	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":6,"version":"go1.14.12"}}}
2021-02-18T09:25:07.443-0500	INFO	[beat]	instance/beat.go:997	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-02-18T08:20:31-05:00","containerized":false,"name":"serverelk-VirtualBox","ip":["127.0.0.1/8","::1/128","192.168.100.8/24","fe80::e55d:ae56:a53c:5eaf/64"],"kernel_version":"5.0.0-32-generic","mac":["08:00:27:64:04:0a"],"os":{"family":"","platform":"linuxmint","name":"Linux Mint","version":"19.3 (Tricia)","major":19,"minor":3,"patch":0,"codename":"tricia"},"timezone":"-05","timezone_offset_sec":-18000,"id":"aaa642a1efb049fba8d6ca0c8d22ff0a"}}}
2021-02-18T09:25:07.444-0500	INFO	[beat]	instance/beat.go:1026	Process info	{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/etc/heartbeat", "exe": "/usr/share/heartbeat/bin/heartbeat", "name": "heartbeat", "pid": 5134, "ppid": 5133, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-02-18T09:25:07.270-0500"}}}
2021-02-18T09:25:07.444-0500	INFO	instance/beat.go:299	Setup Beat: heartbeat; Version: 7.10.1
2021-02-18T09:25:07.444-0500	INFO	[index-management]	idxmgmt/std.go:184	Set output.elasticsearch.index to 'efecty' as ILM is enabled.
2021-02-18T09:25:07.445-0500	INFO	eslegclient/connection.go:99	elasticsearch url: http://localhost:9200
2021-02-18T09:25:07.447-0500	INFO	[publisher]	pipeline/module.go:113	Beat name: serverelk-VirtualBox
2021-02-18T09:25:07.447-0500	INFO	instance/beat.go:455	heartbeat start running.
2021-02-18T09:25:07.447-0500	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2021-02-18T09:25:07.447-0500	INFO	beater/heartbeat.go:82	heartbeat is running! Hit CTRL-C to stop it.
2021-02-18T09:25:07.450-0500	INFO	monitors/factory.go:108	Client connection with: beat.ClientConfig{PublishMode:0x0, Processing:beat.ProcessingConfig{EventMetadata:common.EventMetadata{Fields:null, FieldsUnderRoot:false, Tags:[]string(nil)}, Meta:null, Fields:null, DynamicFields:(*common.MapStrPointer)(nil), Processor:beat.ProcessorList(nil), KeepNull:false, DisableHost:false, Private:interface {}(nil)}, CloseRef:beat.CloseRef(nil), WaitClose:0, ACKHandler:beat.ACKer(nil), Events:beat.ClientEventer(nil)}
2021-02-18T09:25:07.450-0500	INFO	cfgfile/reload.go:164	Config reloader started
2021-02-18T09:25:08.468-0500	INFO	[publisher_pipeline_output]	pipeline/output.go:143	Connecting to backoff(elasticsearch(http://localhost:9200))
2021-02-18T09:25:08.470-0500	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2021-02-18T09:25:08.470-0500	INFO	[publisher]	pipeline/retry.go:223	  done
2021-02-18T09:25:08.480-0500	INFO	[esclientleg]	eslegclient/connection.go:314	Attempting to connect to Elasticsearch version 7.10.1
2021-02-18T09:25:08.523-0500	INFO	[license]	licenser/es_callback.go:51	Elasticsearch license: Basic
2021-02-18T09:25:08.526-0500	INFO	[esclientleg]	eslegclient/connection.go:314	Attempting to connect to Elasticsearch version 7.10.1
2021-02-18T09:25:08.547-0500	INFO	[index-management]	idxmgmt/std.go:261	Auto ILM enable success.
2021-02-18T09:25:08.552-0500	INFO	[index-management.ilm]	ilm/std.go:139	do not generate ilm policy: exists=true, overwrite=false
2021-02-18T09:25:08.552-0500	INFO	[index-management]	idxmgmt/std.go:274	ILM policy successfully loaded.
2021-02-18T09:25:08.552-0500	INFO	[index-management]	idxmgmt/std.go:407	Set setup.template.name to '{efecty {now/d}-000001}' as ILM is enabled.
2021-02-18T09:25:08.552-0500	INFO	[index-management]	idxmgmt/std.go:412	Set setup.template.pattern to 'efecty-*' as ILM is enabled.
2021-02-18T09:25:08.552-0500	INFO	[index-management]	idxmgmt/std.go:446	Set settings.index.lifecycle.rollover_alias in template to {efecty {now/d}-000001} as ILM is enabled.
2021-02-18T09:25:08.553-0500	INFO	[index-management]	idxmgmt/std.go:450	Set settings.index.lifecycle.name in template to {heartbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2021-02-18T09:25:08.611-0500	INFO	template/load.go:117	Try loading template efecty to Elasticsearch
2021-02-18T09:25:08.945-0500	INFO	template/load.go:109	template with name 'efecty' loaded.
2021-02-18T09:25:08.945-0500	INFO	[index-management]	idxmgmt/std.go:298	Loaded index template.
2021-02-18T09:25:09.589-0500	INFO	[index-management]	idxmgmt/std.go:309	Write alias successfully generated.
2021-02-18T09:25:09.591-0500	INFO	[publisher_pipeline_output]	pipeline/output.go:151	Connection to backoff(elasticsearch(http://localhost:9200)) established
2021-02-18T09:25:12.453-0500	INFO	monitors/factory.go:108	Client connection with: beat.ClientConfig{PublishMode:0x0, Processing:beat.ProcessingConfig{EventMetadata:common.EventMetadata{Fields:null, FieldsUnderRoot:false, Tags:[]string(nil)}, Meta:null, Fields:null, DynamicFields:(*common.MapStrPointer)(nil), Processor:beat.ProcessorList(nil), KeepNull:false, DisableHost:false, Private:interface {}(nil)}, CloseRef:beat.CloseRef(nil), WaitClose:0, ACKHandler:beat.ACKer(nil), Events:beat.ClientEventer(nil)}
2021-02-18T09:25:37.449-0500	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":90,"time":{"ms":92}},"total":{"ticks":180,"time":{"ms":190},"value":180},"user":{"ticks":90,"time":{"ms":98}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":11},"info":{"ephemeral_id":"7f67cf89-2d10-4bcb-85bd-0ee9c04bcedf","uptime":{"ms":30040}},"memstats":{"gc_next":7742224,"memory_alloc":5403656,"memory_total":23990744,"rss":55209984},"runtime":{"goroutines":17}},"heartbeat":{"http":{"endpoint_starts":1,"monitor_starts":1},"icmp":{"endpoint_starts":1,"monitor_starts":1}},"libbeat":{"config":{"module":{"running":1,"starts":1},"reloads":1,"scans":5},"output":{"events":{"acked":4,"batches":4,"total":4},"read":{"bytes":4585},"type":"elasticsearch","write":{"bytes":57235}},"pipeline":{"clients":2,"events":{"active":0,"published":4,"retry":1,"total":4},"queue":{"acked":4}}},"system":{"cpu":{"cores":6},"load":{"1":0.9,"15":0.45,"5":0.62,"norm":{"1":0.15,"15":0.075,"5":0.1033}}}}}}
2021-02-18T09:26:07.450-0500	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":110,"time":{"ms":26}},"total":{"ticks":210,"time":{"ms":33},"value":210},"user":{"ticks":100,"time":{"ms":7}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":11},"info":{"ephemeral_id":"7f67cf89-2d10-4bcb-85bd-0ee9c04bcedf","uptime":{"ms":60039}},"memstats":{"gc_next":8278560,"memory_alloc":4162296,"memory_total":25548632,"rss":245760},"runtime":{"goroutines":17}},"libbeat":{"config":{"module":{"running":1},"scans":6},"output":{"events":{"acked":3,"batches":3,"total":3},"read":{"bytes":985},"write":{"bytes":5135}},"pipeline":{"clients":2,"events":{"active":0,"published":3,"total":3},"queue":{"acked":3}}},"system":{"load":{"1":0.82,"15":0.46,"5":0.62,"norm":{"1":0.1367,"15":0.0767,"5":0.1033}}}}}}

I am wondering where this comes from:

Set setup.template.name to '{efecty {now/d}-000001}' as ILM is enabled.

I haven't seen it in your config and this should not be the case - this should normally be the rollover alias. Is it possible that your Yaml file is invalid - maybe there is a leading blank before the setup keys?

yes, I already corrected it and then I looked at the index management and I got the index.

but it will only be done through the debug mode for the service to work?

No, the debug mode is only to find the root cause of the issue.

1 Like

ok, very thanks!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.