Heartbeat to monitor AWS Elastic Beanstalk

Elastic Stack Beats
1

Hi everyone,

Background Info:
I am working on a project that requires me to monitor the health of AWS Elastic Beanstalk (EB) applications, but when I use the URL provided for my applications by AWS EB, I keep seeing the Endpoint status as down. However, if I use google.com for example, it works fine.

Here is a picture of my Kibana Uptime output:
You can find it by going to Kibana > Uptime.

image
image.png1827×688 59 KB

Question:

  1. Do you think the problem is from my configuration file or from AWS EB side regarding permissions, etc?
  2. By any chance do you think it might be related to not enabling the 'AWS Elastic Beanstalk Enhanced Health Reporting'?
    https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/health-enhanced-enable.html?icmpid=docs_elasticbeanstalk_console
  3. Any debugging strategy I can use to find out the problem?

Configuration files:
Note: I have not changed the files under monitors.d directory such as sample.http.yml .

heartbeat.yml:

heartbeat.config.monitors:
  path: ${path.config}/monitors.d/*.yml
  reload.enabled: false
  reload.period: 5s

heartbeat.monitors:
- type: http
  urls: ["***.elasticbeanstalk.com", "http://google.com"]
  schedule: '@every 10s'
  #timeout: 16s

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false

setup.kibana:
  host: "localhost:5601"

output.elasticsearch:
  hosts: ["localhost:9200"]

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
2

What specific errors is heartbeat returning? They should be visible on the errors list in the ap.

3

@Andrew_Cholakian1

I didn't observe any errors, but here is my outupt from Ubuntu Command Line:

$ ./heartbeat -e -c heartbeat.yml
    2019-06-18T08:05:45.252-0700	INFO	instance/beat.go:571	Home path: [/home/kourosh/Documents/heartbeatBeanstalk] Config path: [/home/kourosh/Documents/heartbeatBeanstalk] Data path: [/home/kourosh/Documents/heartbeatBeanstalk/data] Logs path: [/home/kourosh/Documents/heartbeatBeanstalk/logs]
    2019-06-18T08:05:45.252-0700	INFO	instance/beat.go:579	Beat ID: 32106479-86f6-4272-92e8-6fa1cd94c04c
    2019-06-18T08:05:45.253-0700	INFO	[index-management.ilm]	ilm/ilm.go:129	Policy name: heartbeat-7.1.1
    2019-06-18T08:05:45.258-0700	INFO	[seccomp]	seccomp/seccomp.go:116	Syscall filter successfully installed
    2019-06-18T08:05:45.258-0700	INFO	[beat]	instance/beat.go:827	Beat info	{"system_info": {"beat": {"path": {"config": "/home/kourosh/Documents/heartbeatBeanstalk", "data": "/home/kourosh/Documents/heartbeatBeanstalk/data", "home": "/home/kourosh/Documents/heartbeatBeanstalk", "logs": "/home/kourosh/Documents/heartbeatBeanstalk/logs"}, "type": "heartbeat", "uuid": "32106479-86f6-4272-92e8-6fa1cd94c04c"}}}
    2019-06-18T08:05:45.259-0700	INFO	[beat]	instance/beat.go:836	Build info	{"system_info": {"build": {"commit": "3358d9a5a09e3c6709a2d3aaafde628ea34e8419", "libbeat": "7.1.1", "time": "2019-05-23T13:07:32.000Z", "version": "7.1.1"}}}
    2019-06-18T08:05:45.259-0700	INFO	[beat]	instance/beat.go:839	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.11.5"}}}
    2019-06-18T08:05:45.262-0700	INFO	[beat]	instance/beat.go:843	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-06-05T19:21:49-07:00","containerized":false,"name":"itecoation-Lenovo-ideapad-330S-15IKB","ip":["127.0.0.1/8","::1/128","192.168.43.66/24","fe80::15ac:257f:5731:6bed/64","172.17.0.1/16","fe80::42:c8ff:fe0a:34b1/64","172.18.0.1/16","fe80::42:f0ff:fe00:bfc2/64","fe80::a81a:96ff:fec3:cc7b/64"],"kernel_version":"4.18.0-20-generic","mac":["3c:91:80:26:3c:29","02:42:c8:0a:34:b1","02:42:f0:00:bf:c2","aa:1a:96:c3:cc:7b"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"18.10 (Cosmic Cuttlefish)","major":18,"minor":10,"patch":0,"codename":"cosmic"},"timezone":"PDT","timezone_offset_sec":-25200,"id":"41f287fd57c74f4c8a7949c51f54bbbc"}}}
    2019-06-18T08:05:45.263-0700	INFO	[beat]	instance/beat.go:872	Process info	{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/home/kourosh/Documents/heartbeatBeanstalk", "exe": "/home/kourosh/Documents/heartbeatBeanstalk/heartbeat", "name": "heartbeat", "pid": 31166, "ppid": 743, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2019-06-18T08:05:44.690-0700"}}}
    2019-06-18T08:05:45.263-0700	INFO	instance/beat.go:280	Setup Beat: heartbeat; Version: 7.1.1
    2019-06-18T08:05:45.263-0700	INFO	[index-management]	idxmgmt/std.go:165	Set output.elasticsearch.index to 'heartbeat-7.1.1' as ILM is enabled.
    2019-06-18T08:05:45.264-0700	INFO	elasticsearch/client.go:165	Elasticsearch url: http://localhost:9200
    2019-06-18T08:05:45.264-0700	INFO	[publisher]	pipeline/module.go:97	Beat name: itecoation-Lenovo-ideapad-330S-15IKB
    2019-06-18T08:05:45.264-0700	INFO	instance/beat.go:391	heartbeat start running.
    2019-06-18T08:05:45.264-0700	INFO	beater/heartbeat.go:80	heartbeat is running! Hit CTRL-C to stop it.
    2019-06-18T08:05:45.264-0700	INFO	[monitoring]	log/log.go:117	Starting metrics logging every 30s
    2019-06-18T08:05:45.266-0700	INFO	cfgfile/reload.go:150	Config reloader started
    2019-06-18T08:05:45.266-0700	INFO	cfgfile/reload.go:205	Loading of config files completed.
    2019-06-18T08:05:48.258-0700	INFO	add_cloud_metadata/add_cloud_metadata.go:346	add_cloud_metadata: hosting provider type not detected.
    2019-06-18T08:06:05.266-0700	INFO	scheduler/scheduler.go:334	Scheduled job 'auto-http-0XF0F5A69BE79298A5' already active.
    2019-06-18T08:06:05.266-0700	INFO	scheduler/scheduler.go:334	Scheduled job 'auto-http-0XF0F5A69BE79298A5' already active.
    2019-06-18T08:06:06.281-0700	INFO	pipeline/output.go:95	Connecting to backoff(elasticsearch(http://localhost:9200))
    2019-06-18T08:06:06.339-0700	INFO	[index-management.ilm]	ilm/std.go:134	do not generate ilm policy: exists=true, overwrite=false
    2019-06-18T08:06:06.339-0700	INFO	[index-management]	idxmgmt/std.go:366	Set setup.template.pattern to 'heartbeat-7.1.1-*' as ILM is enabled.
    2019-06-18T08:06:06.339-0700	INFO	[index-management]	idxmgmt/std.go:400	Set settings.index.lifecycle.rollover_alias in template to {heartbeat-7.1.1 {now/d}-000001} as ILM is enabled.
    2019-06-18T08:06:06.340-0700	INFO	[index-management]	idxmgmt/std.go:404	Set settings.index.lifecycle.name in template to {heartbeat-7.1.1 map[policy:{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}]} as ILM is enabled.
    2019-06-18T08:06:06.341-0700	INFO	template/load.go:129	Template already exists and will not be overwritten.
    2019-06-18T08:06:06.342-0700	INFO	pipeline/output.go:105	Connection to backoff(elasticsearch(http://localhost:9200)) established
    2019-06-18T08:06:15.270-0700	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":30,"time":{"ms":39}},"total":{"ticks":100,"time":{"ms":115},"value":100},"user":{"ticks":70,"time":{"ms":76}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":8},"info":{"ephemeral_id":"7dd75f1c-e2e4-479d-8a76-2449d178febc","uptime":{"ms":30037}},"memstats":{"gc_next":4194304,"memory_alloc":2071520,"memory_total":4831144,"rss":24629248}},"heartbeat":{"http":{"endpoint_starts":2,"monitor_starts":1}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"acked":2,"batches":1,"total":2},"read":{"bytes":2245},"type":"elasticsearch","write":{"bytes":3256}},"pipeline":{"clients":2,"events":{"active":0,"published":2,"retry":2,"total":2},"queue":{"acked":2}}},"system":{"cpu":{"cores":8},"load":{"1":1.57,"15":1.27,"5":1.15,"norm":{"1":0.1963,"15":0.1588,"5":0.1438}}}}}}
4

My administrator told me this might be due to the fact that I am not part of the same AWS VPC. I'm waiting for them to add me, and will get back to this post if anything changes. In the meantime, feel free to add your suggestions, it might be helpful to someone in the future.

5

Hope that fixed it. The errors, by the way, would be in the UI, not in the logs

6

Yes, other links work fine, it was just Elastic Beanstalk which I didn't have access rights to monitor its uptime monitoring.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.