Metricbeat + AWS Module: runtime error: index out of range [0] with length 0

I'm trying to configure the AWS module but when I try to start the metricbeat, I receive this error:

FATAL [metricbeat] instance/beat.go:164 Failed due to panic

This error are not "too clear" and I started my metricbeat with debug logging level. The log level are not clear about what are happening.

I checked all the configs that I know, but the problem persist.

I removed all the modules (was only aws and system) and test only with aws. If I disable aws module, the metricbeat works.

Can someone help me?

metricbeat.yml

logging.level: debug
metricbeat.modules:
- module: aws
  metricsets: 
  - billing
  period: 12h

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false
setup.ilm:
    overwrite: true

output.elasticsearch:
  hosts: ["elasticsearch-url:9200"]
  username: "username"
  password: "password"

setup.kibana:
  host: "elasticsearch-url:5601"
  username: "username"
  password: "password"
  ssl.verification_mode: none
  protocol: "https"

Metricbeat version:

# metricbeat version
metricbeat version 7.9.1 (amd64), libbeat 7.9.1 [ad823eca4cc74439d1a44351c596c12ab51054f5 built 2020-09-01 20:03:55 +0000 UTC]

Complete output of debug:

# metricbeat -c metricbeat.yml -e -d "*"
2020-09-14T20:34:30.662-0300	INFO	instance/beat.go:640	Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat]
2020-09-14T20:34:30.662-0300	DEBUG	[beat]	instance/beat.go:692	Beat metadata path: /var/lib/metricbeat/meta.json
2020-09-14T20:34:30.662-0300	INFO	instance/beat.go:648	Beat ID: ad2b2e0b-e8b5-4cb3-81a2-b6199b6eec7f
2020-09-14T20:34:30.662-0300	DEBUG	[seccomp]	seccomp/seccomp.go:117	Loading syscall filter	{"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","chown","clock_gettime","clone","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchmodat","fchown","fchownat","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}}}
2020-09-14T20:34:30.663-0300	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2020-09-14T20:34:30.663-0300	INFO	[beat]	instance/beat.go:976	Beat info	{"system_info": {"beat": {"path": {"config": "/etc/metricbeat", "data": "/var/lib/metricbeat", "home": "/usr/share/metricbeat", "logs": "/var/log/metricbeat"}, "type": "metricbeat", "uuid": "ad2b2e0b-e8b5-4cb3-81a2-b6199b6eec7f"}}}
2020-09-14T20:34:30.663-0300	INFO	[beat]	instance/beat.go:985	Build info	{"system_info": {"build": {"commit": "ad823eca4cc74439d1a44351c596c12ab51054f5", "libbeat": "7.9.1", "time": "2020-09-01T20:03:55.000Z", "version": "7.9.1"}}}
2020-09-14T20:34:30.663-0300	INFO	[beat]	instance/beat.go:988	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.14.7"}}}
2020-09-14T20:34:30.663-0300	INFO	[beat]	instance/beat.go:992	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-09-14T20:03:54-03:00","containerized":false,"name":"ip-172-29-39-170.us-east-1.compute.internal","ip":["127.0.0.1/8","172.29.39.170/22"],"kernel_version":"4.14.193-149.317.amzn2.x86_64","mac":["16:83:7c:6a:e5:5b"],"os":{"family":"redhat","platform":"amzn","name":"Amazon Linux","version":"2","major":2,"minor":0,"patch":0,"codename":"Karoo"},"timezone":"-03","timezone_offset_sec":-10800,"id":"ec2eb2a972ed3b680c44e709588d1e20"}}}
2020-09-14T20:34:30.664-0300	INFO	[beat]	instance/beat.go:1021	Process info	{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/etc/metricbeat", "exe": "/usr/share/metricbeat/bin/metricbeat", "name": "metricbeat", "pid": 5467, "ppid": 4955, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-09-14T20:34:30.200-0300"}}}
2020-09-14T20:34:30.664-0300	INFO	instance/beat.go:299	Setup Beat: metricbeat; Version: 7.9.1
2020-09-14T20:34:30.664-0300	DEBUG	[beat]	instance/beat.go:325	Initializing output plugins
2020-09-14T20:34:30.664-0300	INFO	[index-management]	idxmgmt/std.go:184	Set output.elasticsearch.index to 'metricbeat-7.9.1' as ILM is enabled.
2020-09-14T20:34:30.664-0300	INFO	eslegclient/connection.go:99	elasticsearch url: http://xxxxxxx:9200
2020-09-14T20:34:30.664-0300	DEBUG	[publisher]	pipeline/consumer.go:148	start pipeline event consumer
2020-09-14T20:34:30.664-0300	INFO	[publisher]	pipeline/module.go:113	Beat name: ip-xxxxx.us-east-1.compute.internal
2020-09-14T20:34:30.675-0300	DEBUG	[modules]	beater/metricbeat.go:151	Available modules and metricsets: Register [ModuleFactory:[aws, azure, beat, cloudfoundry, docker, elasticsearch, kibana, linux, logstash, mongodb, mssql, mysql, oracle, postgresql, system, uwsgi], MetricSetFactory:[aerospike/namespace, apache/status, appsearch/stats, aws/cloudwatch, aws/ec2, aws/rds, aws/s3_daily_storage, aws/s3_request, aws/sqs, azure/app_insights, azure/billing, azure/compute_vm, azure/compute_vm_scaleset, azure/monitor, azure/storage, beat/state, beat/stats, ceph/cluster_disk, ceph/cluster_health, ceph/cluster_status, ceph/mgr_cluster_disk, ceph/mgr_cluster_health, ceph/mgr_osd_perf, ceph/mgr_osd_pool_stats, ceph/mgr_osd_tree, ceph/mgr_pool_disk, ceph/monitor_health, ceph/osd_df, ceph/osd_tree, ceph/pool_disk, cloudfoundry/container, cloudfoundry/counter, cloudfoundry/value, consul/agent, coredns/stats, couchbase/bucket, couchbase/cluster, couchbase/node, couchdb/server, docker/container, docker/cpu, docker/diskio, docker/event, docker/healthcheck, docker/image, docker/info, docker/memory, docker/network, dropwizard/collector, elasticsearch/ccr, elasticsearch/cluster_stats, elasticsearch/enrich, elasticsearch/index, elasticsearch/index_recovery, elasticsearch/index_summary, elasticsearch/ml_job, elasticsearch/node, elasticsearch/node_stats, elasticsearch/pending_tasks, elasticsearch/shard, envoyproxy/server, etcd/leader, etcd/metrics, etcd/self, etcd/store, golang/expvar, golang/heap, googlecloud/metrics, graphite/server, haproxy/info, haproxy/stat, http/json, http/server, istio/citadel, istio/galley, istio/mesh, istio/mixer, istio/pilot, jolokia/jmx, kafka/consumergroup, kafka/partition, kibana/stats, kibana/status, kubernetes/apiserver, kubernetes/container, kubernetes/controllermanager, kubernetes/event, kubernetes/node, kubernetes/pod, kubernetes/proxy, kubernetes/scheduler, kubernetes/state_container, kubernetes/state_cronjob, kubernetes/state_deployment, kubernetes/state_node, kubernetes/state_persistentvolume, kubernetes/state_persistentvolumeclaim, kubernetes/state_pod, kubernetes/state_replicaset, kubernetes/state_resourcequota, kubernetes/state_service, kubernetes/state_statefulset, kubernetes/state_storageclass, kubernetes/system, kubernetes/volume, kvm/dommemstat, kvm/status, linux/conntrack, linux/ksm, linux/pageinfo, logstash/node, logstash/node_stats, memcached/stats, mongodb/collstats, mongodb/dbstats, mongodb/metrics, mongodb/replstatus, mongodb/status, mssql/performance, mssql/transaction_log, munin/node, mysql/galera_status, mysql/query, mysql/status, nats/connections, nats/routes, nats/stats, nats/subscriptions, nginx/stubstatus, openmetrics/collector, oracle/performance, oracle/tablespace, php_fpm/pool, php_fpm/process, postgresql/activity, postgresql/bgwriter, postgresql/database, postgresql/statement, prometheus/collector, prometheus/query, prometheus/remote_write, rabbitmq/connection, rabbitmq/exchange, rabbitmq/node, rabbitmq/queue, redis/info, redis/key, redis/keyspace, sql/query, stan/channels, stan/stats, stan/subscriptions, statsd/server, system/core, system/cpu, system/diskio, system/entropy, system/filesystem, system/fsstat, system/load, system/memory, system/network, system/network_summary, system/process, system/process_summary, system/raid, system/service, system/socket, system/socket_summary, system/uptime, system/users, traefik/health, uwsgi/status, vsphere/datastore, vsphere/host, vsphere/virtualmachine, zookeeper/connection, zookeeper/mntr, zookeeper/server], LightModules:[LightModules:[activemq/queue, activemq/topic, activemq/broker, mysql/performance, cockroachdb/status, azure/container_registry, azure/container_instance, azure/container_service, azure/database_account, ibmmq/qmgr, aws/usage, aws/vpn, aws/transitgateway, aws/natgateway, aws/elb, aws/ebs, aws/lambda, aws/dynamodb, aws/billing, aws/sns, googlecloud/compute, googlecloud/pubsub, googlecloud/loadbalancing, googlecloud/storage, tomcat/threading, tomcat/memory, tomcat/requests, tomcat/cache, iis/webserver, iis/website, kafka/broker, kafka/producer, kafka/consumer, redisenterprise/node, redisenterprise/proxy]]]
2020-09-14T20:34:30.677-0300	DEBUG	[get_aws_credentials]	aws/credentials.go:74	Using shared credential profile for AWS credential
2020-09-14T20:34:30.678-0300	DEBUG	[aws.billing]	aws/aws.go:93	Metricset level config for period: 12h0m0s
2020-09-14T20:34:30.678-0300	DEBUG	[aws.billing]	aws/aws.go:94	Metricset level config for tags filter: []
2020-09-14T20:34:30.762-0300	INFO	runtime/panic.go:975	metricbeat stopped.
2020-09-14T20:34:30.762-0300	FATAL	[metricbeat]	instance/beat.go:164	Failed due to panic.	{"panic": "runtime error: index out of range [0] with length 0", "stack": "github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1.1\n\t/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:165\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:969\nruntime.goPanicIndex\n\t/usr/local/go/src/runtime/panic.go:88\ngithub.com/elastic/beats/v7/x-pack/metricbeat/module/aws.NewMetricSet\n\t/go/src/github.com/elastic/beats/x-pack/metricbeat/module/aws/aws.go:122\ngithub.com/elastic/beats/v7/x-pack/metricbeat/module/aws/cloudwatch.New\n\t/go/src/github.com/elastic/beats/x-pack/metricbeat/module/aws/cloudwatch/cloudwatch.go:113\ngithub.com/elastic/beats/v7/metricbeat/mb.(*LightMetricSet).Registration.func1\n\t/go/src/github.com/elastic/beats/metricbeat/mb/lightmetricset.go:96\ngithub.com/elastic/beats/v7/metricbeat/mb.initMetricSets\n\t/go/src/github.com/elastic/beats/metricbeat/mb/builders.go:139\ngithub.com/elastic/beats/v7/metricbeat/mb.NewModule\n\t/go/src/github.com/elastic/beats/metricbeat/mb/builders.go:65\ngithub.com/elastic/beats/v7/metricbeat/mb/module.(*Factory).Create\n\t/go/src/github.com/elastic/beats/metricbeat/mb/module/factory.go:44\ngithub.com/elastic/beats/v7/metricbeat/beater.newMetricbeat\n\t/go/src/github.com/elastic/beats/metricbeat/beater/metricbeat.go:169\ngithub.com/elastic/beats/v7/metricbeat/beater.Creator.func1\n\t/go/src/github.com/elastic/beats/metricbeat/beater/metricbeat.go:81\ngithub.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createBeater\n\t/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:359\ngithub.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch\n\t/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:417\ngithub.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1\n\t/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:189\ngithub.com/elastic/beats/v7/libbeat/cmd/instance.Run\n\t/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:190\ngithub.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1\n\t/go/src/github.com/elastic/beats/libbeat/cmd/run.go:36\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:766\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:852\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:800\nmain.main\n\t/go/src/github.com/elastic/beats/x-pack/metricbeat/main.go:21\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"}

Nice finding. Could you please an issue for Beats on Github? It looks like a valid report.

Thank you, @mtojek. I did. Looks like a bug.

I updated to 7.9.2 and the problem are not happens anymore.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.