Start-Service heartbeat fails

ppic · January 20, 2023, 2:16pm

Hello,
I have installed heartbeat on a Windows server.
When I start the service, it stops immediately.

with the command : "Start-Service heartbeat"
it says nothing, but I can see with the command "sc.exe query heartbeat" that it is STOPPED
with the command: "net start heartbeat", it ouputs :

The heartbeat service is starting.
The heartbeat service could not be started.

A system error has occurred.

System error 1067 has occurred.

The process terminated unexpectedly.

And the heartbeat log file in \ProgramData\heartbeat\logs ends with :

ERROR instance/beat.go:1014 Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory C:\Program Files\heartbeat\kibana: No directory C:\Program Files\heartbeat\kibana/7

I am using Kibana and beats v 7.17.6

Before, I have installed filebeat and metricbeat that work well.
(this is my first topic, so excuse possible formatting errors....)

Thanks for the advice

Rios · January 20, 2023, 3:48pm

Have you started PS in admin mode before installation?

Error importing Kibana dashboards:

Have you set in winlogbeat.yml:

setup.kibana:
   host: "localhost:5601"

ppic · January 20, 2023, 4:08pm

I have started in admin mode,
but I have no file winlogbeat.yml

Rios · January 20, 2023, 5:22pm

Sorry heartbeat.yml, and kibana must be reachable from HB host.
Have you add this?

setup.kibana:
   host: "localhost:5601" # put yout kibana host

ppic · January 23, 2023, 9:55am

In heartbeat.yml, I have the line :
host: "https://the_kibana_host:9243"
Same line as in metricbeat.yml and filebeat.yml, that works

Rios · January 23, 2023, 10:19am

OK, have you provided certificates?
If you have everything the same, and you imported dashboards also for Metricbeat, then only problem might be a firewall. If you hadn't imported MB dashboards, then something is there. The params output.elasticsearch or output.logstash are connections for data transfer, not for Kibana.

ppic · January 23, 2023, 10:54am

I did nothing with certificates (neither for metricbeat nor filebeat)
In Kibana, I used dashbords from filebeat and metricbeat.

The error message says :
ERROR instance/beat.go:1014 Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory C:\Program Files\heartbeat\kibana: No directory C:\Program Files\heartbeat\kibana/7

Indeed, the directory C:\Program Files\heartbeat\kibana: is empty
unlike filebeat and metricbeat directories, containing a directory "7" and several other sub-dir and files

I installed heartbeat with : heartbeat-7.17.6-windows-x86_64.zip
from: Heartbeat 7.17.6 | Elastic

It seems inconsistent : the zip has and empty directory heartbeat\kibana, and the setup fails because it is empty.

Do you find it normal ?

Rios · January 23, 2023, 1:32pm

# As of version 7.x sample dashboards are not installed by default but can be found in https://github.com/elastic/uptime-contrib

ppic · January 23, 2023, 3:37pm

Do you mean that the directory C:\Program Files\heartbeat\kibana\7 has to be created and populated by ourselves (contrary to filebeat, metricbeat,.) ?

If so, how can I use this file.ndjson from Github ?

Rios · January 23, 2023, 6:34pm

Manually in Kibana.
You can install the dashboards hosted in this repo using the Management > Saved objects page

ppic · January 24, 2023, 2:35pm

From the Kibana home page, I clicked on " Stack Management" and found the menu item "Saved Objects" with the "Import" item.

I imported the file from Github. Kibana displayed :

Import saved objects

8 objects imported

8 new

Heartbeat HTTP monitoring
heartbeat-*
Heartbeat HTTP pings
HTTP duration heatmap
HTTP monitors
HTTP monitors status
HTTP ping times
HTTP up status

But the folder C:\Program Files\heartbeat\kibana is still empty
and the service still fails to start, with the same :

ERROR instance/beat.go:1014 Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory C:\Program Files\heartbeat\kibana: No directory C:\Program Files\heartbeat\kibana/7

Some more idea ?
Thank you.

Rios · January 25, 2023, 9:02am

I couldn't reproduce this error after many tries, on 8.6, maybe because I have set correctly 1st time.
Not sure, how is your configuration set, HB is pretty simple, did you set user/pass for Kibana:

setup.kibana:
  host: "localhost:5601"
  username: "elastic"
  password: "pass"

If this doesn't work set logging.level: debug and restart the service, maybe will be a trace
The last option is to copy http_dashboard.ndjson in the directory: C:\Program Files\heartbeat\kibana/7

mazoutte · January 25, 2023, 10:43am

Hello,

If you imported the objects from the stack management menu, then you could get rid of the "setup.kibana" section in the heartbeat.yml. (by commenting the section or deleting it).

I'm not using this section in any *beat deployments , since I don't use the default/example dashboards.

Edit : however be careful if you need the proper fields mapping/index template.
my Beat clients don't have the connectivity to Kibana Endpoints ; so I need to use the method you mentioned with importing the needed objects for a first deployment / index template creation.

Rios · January 25, 2023, 10:47am

Good point mazoutte. Yes, I forgot to mention, tray also without setup.kibana.

ppic · January 25, 2023, 11:13am

The section : setup.kibana: with host, username, password is the same as for filebeat and metricbeat .

I set logging.level: debug
The log file in programData ends with :

2023-01-25T10:59:50.460Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:237	Initialize the Kibana 7.17.1 loader
2023-01-25T10:59:50.460Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:237	Kibana URL https://edaas-noprod.kb.elasticaas.ocb.equant.com:9243
2023-01-25T10:59:51.358Z	DEBUG	[Imported multipart file to %s with params %v]	kibana/client.go:376	/api/saved_objects/_import%!(EXTRA url.Values=map[overwrite:[true]])
2023-01-25T10:59:51.358Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:237	Importing directory C:\Program Files\heartbeat\kibana/7
2023-01-25T10:59:51.358Z	DEBUG	[dashboards]	dashboards/kibana_loader.go:237	Import directory C:\Program Files\heartbeat\kibana/7
2023-01-25T10:59:51.376Z	INFO	[monitoring]	log/log.go:192	Total metrics	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":265,"time":{"ms":281}},"total":{"ticks":780,"time":{"ms":796},"value":0},"user":{"ticks":515,"time":{"ms":515}}},"handles":{"open":158},"info":{"ephemeral_id":"994dfb59-dbdb-4d43-9a2e-037914fd2e86","uptime":{"ms":2656},"version":"7.17.6"},"memstats":{"gc_next":21710344,"memory_alloc":16393408,"memory_sys":36482472,"memory_total":48664944,"rss":55656448},"runtime":{"goroutines":22}},"heartbeat":{"browser":{"endpoint_starts":0,"endpoint_stops":0,"monitor_starts":0,"monitor_stops":0},"http":{"endpoint_starts":0,"endpoint_stops":0,"monitor_starts":0,"monitor_stops":0},"icmp":{"endpoint_starts":0,"endpoint_stops":0,"monitor_starts":0,"monitor_stops":0},"scheduler":{"jobs":{"active":0,"missed_deadline":0},"tasks":{"active":0,"waiting":0}},"tcp":{"endpoint_starts":0,"endpoint_stops":0,"monitor_starts":0,"monitor_stops":0}},"libbeat":{"config":{"module":{"running":0,"starts":0,"stops":0},"reloads":0,"scans":0},"output":{"events":{"acked":0,"active":0,"batches":0,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":0},"read":{"bytes":0,"errors":0},"type":"elasticsearch","write":{"bytes":0,"errors":0}},"pipeline":{"clients":0,"events":{"active":0,"dropped":0,"failed":0,"filtered":0,"published":0,"retry":0,"total":0},"queue":{"acked":0,"max_events":4096}}},"system":{"cpu":{"cores":2}}}}}
2023-01-25T10:59:51.376Z	INFO	[monitoring]	log/log.go:193	Uptime: 2.6668001s
2023-01-25T10:59:51.376Z	INFO	[monitoring]	log/log.go:160	Stopping metrics logging.
2023-01-25T10:59:51.377Z	INFO	instance/beat.go:489	heartbeat stopped.
2023-01-25T10:59:51.380Z	ERROR	instance/beat.go:1014	Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory C:\Program Files\heartbeat\kibana: Failed to import dashboard: The directory C:\Program Files\heartbeat\kibana/7 is empty, nothing to import

I copied the file http_dashboard.ndjson in the directory: C:\Program Files\heartbeat\kibana\7
the service failed in the same way.

ppic · January 25, 2023, 11:30am

I commented the "setup.kibana" section and tried to start :

PS C:\Program Files\heartbeat> net start heartbeat
The heartbeat service is starting.
The heartbeat service was started successfully.

It seems better, but the service is not running :

PS C:\Program Files\heartbeat> sc.exe  query  heartbeat

SERVICE_NAME: heartbeat
        TYPE               : 10  WIN32_OWN_PROCESS
        STATE              : 1  STOPPED
        WIN32_EXIT_CODE    : 1067  (0x42b)
        SERVICE_EXIT_CODE  : 0  (0x0)
        CHECKPOINT         : 0x0
        WAIT_HINT          : 0x0

and the log file ends with :

2023-01-25T11:15:17.261Z	INFO	instance/beat.go:489	heartbeat stopped.
2023-01-25T11:15:17.262Z	ERROR	instance/beat.go:1014	Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://localhost:5601/api/status fails: fail to execute the HTTP GET request: Get "http://localhost:5601/api/status": dial tcp [::1]:5601: connectex: No connection could be made because the target machine actively refused it.. Response: .

Rios · January 25, 2023, 12:03pm

Can you copy from heartbeat.yml?

It seems you have set localhost for Kibana instead of https://edaa...quant.com:9243

ppic · January 25, 2023, 1:39pm

It is probably because the section "setup.kibana" was commented, and I guess it took a default value to localhost.

Here is the file heartbeat.yml (with "setup.kibana" enabled, not commented)
(I suppressed most of the commented lines)

############################# Heartbeat ######################################
heartbeat.config.monitors:
  path: ${path.config}/monitors.d/*.yml
  reload.enabled: true
  reload.period: 10s

heartbeat.monitors:
- type: http
  enabled: true
  id: heartbeat_http_check_ID
  service.name: HTTP check with heartbeat
  urls: ["http://10.238.116.238/XConfWebAPI//info/warningCodes"]
  schedule: '@every 20s'

# ======================= Elasticsearch template setting =======================
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

# ================================== General ===================================

# =================================== Kibana ===================================
setup.kibana:
  host: "https://edaas-noprod.kb.elasticaas.ocb.equant.com:9243"
  space.id: "xconf-qa"
  protocol: "https"
  username: "the_username"
  password: "*********"
  ssl.verification_mode: none

# =============================== Elastic Cloud ================================

# ================================== Outputs ===================================
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  hosts: ["edaas-noprod.es.elasticaas.ocb.equant.com:9243"]
  protocol: "https"
  username: "xconf-qa-write"
  password: "Gc3AbM6Jx3"
  ssl.verification_mode: none
  index: "xconf-qa-%{[agent.version]}-%{+yyyy.MM.dd}"

setup.template.name: "xconf-qa"
setup.template.pattern: "xconf-qa-*"
setup.dashboards.index: "xconf-qa-*"
setup.template.enabled: true

setup.ilm.enabled: false

xpack.ml.enabled: false

setup.dashboards.enabled: true

# ------------------------------ Logstash Output -------------------------------

# ================================= Processors =================================
processors:
  - add_observer_metadata:

# ================================== Logging ===================================
logging.level: debug

# ============================= X-Pack Monitoring ==============================

# ============================== Instrumentation ===============================

# ================================= Migration ==================================

Maybe it can help, for comparison : here is the same for metricbeat

###################### Metricbeat Configuration Example #######################

# =========================== Modules configuration ============================

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

# ======================= Elasticsearch template setting =======================
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

# ================================== General ===================================

# ================================= Dashboards =================================

# =================================== Kibana ===================================
setup.kibana:
  host: "https://edaas-noprod.kb.elasticaas.ocb.equant.com:9243"
  space.id: "xconf-qa"
  protocol: "https"
  username: "the_username"
  password: "*********"
  ssl.verification_mode: none

# =============================== Elastic Cloud ================================

# ================================== Outputs ===================================

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  hosts: ["edaas-noprod.es.elasticaas.ocb.equant.com:9243"]
  protocol: "https"
  username: "xconf-qa-write"
  password: "Gc3AbM6Jx3"
  ssl.verification_mode: none
  index: "xconf-qa-%{[agent.version]}-%{+yyyy.MM.dd}"

setup.template.name: "xconf-qa"
setup.template.pattern: "xconf-qa-*"
setup.dashboards.index: "xconf-qa-*"
setup.template.enabled: false

setup.ilm.enabled: false

xpack.ml.enabled: false

# ------------------------------ Logstash Output -------------------------------

# ================================= Processors =================================
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# ================================== Logging ===================================

# ============================= X-Pack Monitoring ==============================

# ============================== Instrumentation ===============================

# ================================= Migration ==================================

ppic · January 25, 2023, 1:48pm

by the way, I compare both and noticed :

in metricbeat.yml: setup.template.enabled: false
in heartbeat.yml: setup.template.enabled: true

Then I set it to false in heartbeat.yml, and I could start the service !!!!

PS C:\Program Files\heartbeat> net start heartbeat
The heartbeat service is starting.
The heartbeat service was started successfully.

PS C:\Program Files\heartbeat> sc.exe  query  heartbeat

SERVICE_NAME: heartbeat
        TYPE               : 10  WIN32_OWN_PROCESS
        STATE              : 4  RUNNING
        (...)

And in Kibana, I could open the dashboard "Heartbeat HTTP monitoring".
(I am now facing index problem that I will try to solve. )

I tink the problem of service startup is solved.... Thank you very much for your time and ideas !

Rios · January 25, 2023, 2:02pm

Can you try with this heartbeat.yml? Set url for testing, ES and Kibana hosts

You will not see data in the Heartbeat HTTP monitoring dashboard because you are using different index name or you have to edit heartbeat-* data view.
After you receive data in the dashoard, then you can play with index name, data view etc.

heartbeat.config.monitors:
  path: ${path.config}/monitors.d/*.yml
  reload.enabled: true
  reload.period: 10s

heartbeat.monitors:
- type: http
  # Set enabled to true (or delete the following line) to enable this example monitor
  enabled: true
  # ID used to uniquely identify this monitor in elasticsearch even if the config changes
  id: heartbeat_http_check_ID
  # Human readable display name for this service in Uptime UI and elsewhere
  name: HTTP check with heartbeat
  # List or urls to query
  urls: ["http://putyourip/XConfWebAPI/info/warningCodes"]

  # Configure task schedule
  schedule: '@every 20s'
  # Total test connection and data exchange timeout
  #timeout: 16s
  
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false
  
#setup.kibana:
#  host: "kibanahost:9243"
#  username: "elastic"
#  password: "pass"
  
output.elasticsearch:
  hosts: ["yourESserver:9243"]
  protocol: "https"
  username: "user"
  password: "pass"
  ssl.verification_mode: none

processors:
  - add_observer_metadata:


logging.level: debug
# logging.to_files: true
# logging.files:
#   path: /path/logs/
#   name: heartbeat.ndjson
#   keepfiles: 7
#   permissions: 0640
#