Metricbeat setup to return localhost metrics

Running 7.11 Elastic on self managed eks 1.19 with Metricbeats 7.9 running on EC2 instance

This is the below configuration of the metricbeat.yaml

metricbeat.modules:
- module: system
  metricsets:
   # - cpu             # CPU usage
    - cpu             # CPU usage
    - filesystem     # filesystem averages
   
- module: http
  metricsets: ["server"]
  period: 10s
  host: ["localhost"]
  port: 8080
  enabled: true

Iam not able to get the http module working. Plan is to capture the response code as per this documentation "HTTP fields | Metricbeat Reference [master] | Elastic" and
HTTP server metricset | Metricbeat Reference [master] | Elastic

The metricbeat is failing to start and Iam not able to get the right config to return the response code. Can someone please help

Please share your Metricbeat logs :slight_smile:

Also please also consider upgrading, 7.15 is latest and you are running slightly older versions.

2021-11-19T22:37:36.716Z	INFO	instance/beat.go:665	Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat]
2021-11-19T22:37:36.717Z	DEBUG	[beat]	instance/beat.go:723	Beat metadata path: /var/lib/metricbeat/meta.json
2021-11-19T22:37:36.717Z	INFO	instance/beat.go:673	Beat ID: b7c91577-81b9-4a1d-a035-0fa2eba5ea93
2021-11-19T22:37:36.717Z	DEBUG	[add_cloud_metadata]	add_cloud_metadata/providers.go:128	add_cloud_metadata: starting to fetch metadata, timeout=3s
2021-11-19T22:37:36.719Z	DEBUG	[add_cloud_metadata]	add_cloud_metadata/providers.go:165	add_cloud_metadata: received disposition for digitalocean after 1.08787ms. result=[provider:digitalocean, error=failed with http status code 404, metadata={}]
2021-11-19T22:37:36.719Z	DEBUG	[add_cloud_metadata]	add_cloud_metadata/providers.go:165	add_cloud_metadata: received disposition for azure after 1.151991ms. result=[provider:azure, error=failed with http status code 404, metadata={}]
2021-11-19T22:37:36.719Z	DEBUG	[add_cloud_metadata]	add_cloud_metadata/providers.go:165	add_cloud_metadata: received disposition for gcp after 1.95014ms. result=[provider:gcp, error=failed with http status code 404, metadata={}]
2021-11-19T22:37:36.720Z	DEBUG	[add_cloud_metadata]	add_cloud_metadata/providers.go:165	add_cloud_metadata: received disposition for aws after 2.13501ms. result=[provider:aws, error=<nil>, metadata={"cloud":{"account":{"id":"622494574705"},"availability_zone":"us-east-1a","image":{"id":"ami-00521671dfc1e0100"},"instance":{"id":"i-0921bed04d4c5ab67"},"machine":{"type":"r5.large"},"provider":"aws","region":"us-east-1","service":{"name":"EC2"}}}]
2021-11-19T22:37:36.720Z	DEBUG	[add_cloud_metadata]	add_cloud_metadata/providers.go:131	add_cloud_metadata: fetchMetadata ran for 2.247235ms
2021-11-19T22:37:36.720Z	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:105	add_cloud_metadata: hosting provider type detected as aws, metadata={"cloud":{"account":{"id":"622494574705"},"availability_zone":"us-east-1a","image":{"id":"ami-00521671dfc1e0100"},"instance":{"id":"i-0921bed04d4c5ab67"},"machine":{"type":"r5.large"},"provider":"aws","region":"us-east-1","service":{"name":"EC2"}}}
2021-11-19T22:37:36.720Z	DEBUG	[processors]	processors/processor.go:120	Generated new processors: add_host_metadata=[netinfo.enabled=[true], cache.ttl=[5m0s]], add_cloud_metadata={"cloud":{"account":{"id":"622494574705"},"availability_zone":"us-east-1a","image":{"id":"ami-00521671dfc1e0100"},"instance":{"id":"i-0921bed04d4c5ab67"},"machine":{"type":"r5.large"},"provider":"aws","region":"us-east-1","service":{"name":"EC2"}}}
2021-11-19T22:37:36.720Z	DEBUG	[seccomp]	seccomp/seccomp.go:117	Loading syscall filter	{"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","chown","clock_gettime","clone","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchmodat","fchown","fchownat","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}}}
2021-11-19T22:37:36.720Z	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2021-11-19T22:37:36.720Z	INFO	[beat]	instance/beat.go:1014	Beat info	{"system_info": {"beat": {"path": {"config": "/etc/metricbeat", "data": "/var/lib/metricbeat", "home": "/usr/share/metricbeat", "logs": "/var/log/metricbeat"}, "type": "metricbeat", "uuid": "b7c91577-81b9-4a1d-a035-0fa2eba5ea93"}}}
2021-11-19T22:37:36.720Z	INFO	[beat]	instance/beat.go:1023	Build info	{"system_info": {"build": {"commit": "e127fc31fc6c00fdf8649808f9421d8f8c28b5db", "libbeat": "7.14.0", "time": "2021-07-29T21:09:05.000Z", "version": "7.14.0"}}}
2021-11-19T22:37:36.720Z	INFO	[beat]	instance/beat.go:1026	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.16.6"}}}
2021-11-19T22:37:36.721Z	INFO	[beat]	instance/beat.go:1030	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-08-16T20:12:39Z","containerized":false,"name":"ip-10-128-0-50.ec2.internal","ip":["127.0.0.1/8","::1/128","10.128.0.50/26","fe80::101b:d7ff:fe53:7d4d/64"],"kernel_version":"3.10.0-1160.15.2.el7.x86_64","mac":["12:1b:d7:53:7d:4d"],"os":{"type":"linux","family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":9,"patch":2009,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"3d5c05376530a2eb49e3e90576f83c5b"}}}
2021-11-19T22:37:36.721Z	INFO	[beat]	instance/beat.go:1059	Process info	{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/", "exe": "/usr/share/metricbeat/bin/metricbeat", "name": "metricbeat", "pid": 21825, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-11-19T22:37:36.420Z"}}}
2021-11-19T22:37:36.721Z	INFO	instance/beat.go:309	Setup Beat: metricbeat; Version: 7.14.0
2021-11-19T22:37:36.721Z	DEBUG	[beat]	instance/beat.go:335	Initializing output plugins
2021-11-19T22:37:36.722Z	DEBUG	[publisher]	pipeline/consumer.go:148	start pipeline event consumer
2021-11-19T22:37:36.722Z	INFO	[publisher]	pipeline/module.go:113	Beat name: ip-10-128-0-50.ec2.internal
2021-11-19T22:37:36.739Z	DEBUG	[modules]	beater/metricbeat.go:151	Available modules and metricsets: Register [ModuleFactory:[aws, awsfargate, azure, beat, cloudfoundry, docker, elasticsearch, kibana, kubernetes, linux, logstash, mongodb, mssql, mysql, oracle, postgresql, system, uwsgi], MetricSetFactory:[aerospike/namespace, apache/status, appsearch/stats, aws/billing, aws/cloudwatch, awsfargate/task_stats, azure/app_insights, azure/billing, azure/monitor, azure/storage, beat/state, beat/stats, ceph/cluster_disk, ceph/cluster_health, ceph/cluster_status, ceph/mgr_cluster_disk, ceph/mgr_cluster_health, ceph/mgr_osd_perf, ceph/mgr_osd_pool_stats, ceph/mgr_osd_tree, ceph/mgr_pool_disk, ceph/monitor_health, ceph/osd_df, ceph/osd_tree, ceph/pool_disk, cloudfoundry/container, cloudfoundry/counter, cloudfoundry/value, consul/agent, coredns/stats, couchbase/bucket, couchbase/cluster, couchbase/node, couchdb/server, docker/container, docker/cpu, docker/diskio, docker/event, docker/healthcheck, docker/image, docker/info, docker/memory, docker/network, docker/network_summary, dropwizard/collector, elasticsearch/ccr, elasticsearch/cluster_stats, elasticsearch/enrich, elasticsearch/index, elasticsearch/index_recovery, elasticsearch/index_summary, elasticsearch/ml_job, elasticsearch/node, elasticsearch/node_stats, elasticsearch/pending_tasks, elasticsearch/shard, envoyproxy/server, etcd/leader, etcd/metrics, etcd/self, etcd/store, gcp/billing, gcp/metrics, golang/expvar, golang/heap, graphite/server, haproxy/info, haproxy/stat, http/json, http/server, istio/citadel, istio/galley, istio/mesh, istio/mixer, istio/pilot, jolokia/jmx, kafka/consumergroup, kafka/partition, kibana/stats, kibana/status, kubernetes/apiserver, kubernetes/container, kubernetes/controllermanager, kubernetes/event, kubernetes/node, kubernetes/pod, kubernetes/proxy, kubernetes/scheduler, kubernetes/state_container, kubernetes/state_cronjob, kubernetes/state_daemonset, kubernetes/state_deployment, kubernetes/state_node, kubernetes/state_persistentvolume, kubernetes/state_persistentvolumeclaim, kubernetes/state_pod, kubernetes/state_replicaset, kubernetes/state_resourcequota, kubernetes/state_service, kubernetes/state_statefulset, kubernetes/state_storageclass, kubernetes/system, kubernetes/volume, kvm/dommemstat, kvm/status, linux/conntrack, linux/iostat, linux/ksm, linux/memory, linux/pageinfo, logstash/node, logstash/node_stats, memcached/stats, mongodb/collstats, mongodb/dbstats, mongodb/metrics, mongodb/replstatus, mongodb/status, mssql/performance, mssql/transaction_log, munin/node, mysql/galera_status, mysql/query, mysql/status, nats/connection, nats/connections, nats/route, nats/routes, nats/stats, nats/subscriptions, nginx/stubstatus, openmetrics/collector, oracle/performance, oracle/tablespace, php_fpm/pool, php_fpm/process, postgresql/activity, postgresql/bgwriter, postgresql/database, postgresql/statement, prometheus/collector, prometheus/query, prometheus/remote_write, rabbitmq/connection, rabbitmq/exchange, rabbitmq/node, rabbitmq/queue, redis/info, redis/key, redis/keyspace, sql/query, stan/channels, stan/stats, stan/subscriptions, statsd/server, system/core, system/cpu, system/diskio, system/entropy, system/filesystem, system/fsstat, system/load, system/memory, system/network, system/network_summary, system/process, system/process_summary, system/raid, system/service, system/socket, system/socket_summary, system/uptime, system/users, traefik/health, uwsgi/status, vsphere/datastore, vsphere/host, vsphere/virtualmachine, zookeeper/connection, zookeeper/mntr, zookeeper/server], LightModules:[LightModules:[mysql/performance, redisenterprise/node, redisenterprise/proxy, kafka/producer, kafka/consumer, kafka/broker, aws/vpn, aws/natgateway, aws/sqs, aws/lambda, aws/dynamodb, aws/rds, aws/s3_daily_storage, aws/s3_request, aws/transitgateway, aws/kinesis, aws/ec2, aws/ebs, aws/usage, aws/sns, aws/elb, ibmmq/qmgr, iis/webserver, iis/website, tomcat/cache, tomcat/threading, tomcat/memory, tomcat/requests, gcp/compute, gcp/pubsub, gcp/loadbalancing, gcp/storage, cockroachdb/status, istio/proxy, istio/istiod, azure/container_registry, azure/container_instance, azure/container_service, azure/database_account, azure/app_state, azure/compute_vm, azure/compute_vm_scaleset, googlecloud/compute, googlecloud/pubsub, googlecloud/loadbalancing, googlecloud/storage, activemq/broker, activemq/queue, activemq/topic]]]
2021-11-19T22:37:36.739Z	INFO	filesystem/filesystem.go:57	Ignoring filesystem types: squashfs, nsfs, nfs, smbfs, autofs
2021-11-19T22:37:36.740Z	INFO	instance/beat.go:442	metricbeat stopped.
2021-11-19T22:37:36.741Z	ERROR	instance/beat.go:989	Exiting: 1 error: can not convert 'object' into 'string' accessing 'metricbeat.modules.1.host' (source:'/etc/metricbeat/metricbeat.yml')

Can you please share your entire metricbeat.yml.

Updated above

If you don't want to put the port directly in the host name (eg hosts: ["localhost:80"] ) then your port line needs to be quoted too (eg port: "80" )

Thanks the port change didnt help the Metric beat still doesnt return any http metrics

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.