Metricbeat "Failed due to panic.{"panic": "runtime error: index out of range [0]"

Elastic Stack Beats
1

I am using the Metricbeat AWS module while starting Metricbeat, I am facing the following error. I went through Git issues but seems it's fixed in the version I am using. I am pasting my aws.yml as well for reference.

Error

2021-03-12T07:36:48.222Z	INFO	instance/beat.go:640	Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat]
2021-03-12T07:36:48.223Z	INFO	instance/beat.go:648	Beat ID: 525ded62-ce68-4a71-85d4-3e09f56afc20
2021-03-12T07:36:48.227Z	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:93	add_cloud_metadata: hosting provider type detected as aws, metadata={"account":{"id":"917006514933"},"availability_zone":"ap-south-1a","image":{"id":"ami-042b34111b1289ccd"},"instance":{"id":"i-0f790ff5d45cffd48"},"machine":{"type":"t2.micro"},"provider":"aws","region":"ap-south-1"}
2021-03-12T07:36:48.227Z	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2021-03-12T07:36:48.228Z	INFO	[beat]	instance/beat.go:976	Beat info	{"system_info": {"beat": {"path": {"config": "/etc/metricbeat", "data": "/var/lib/metricbeat", "home": "/usr/share/metricbeat", "logs": "/var/log/metricbeat"}, "type": "metricbeat", "uuid": "525ded62-ce68-4a71-85d4-3e09f56afc20"}}}
2021-03-12T07:36:48.228Z	INFO	[beat]	instance/beat.go:985	Build info	{"system_info": {"build": {"commit": "ad823eca4cc74439d1a44351c596c12ab51054f5", "libbeat": "7.9.1", "time": "2020-09-01T20:03:55.000Z", "version": "7.9.1"}}}
2021-03-12T07:36:48.228Z	INFO	[beat]	instance/beat.go:988	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":1,"version":"go1.14.7"}}}
2021-03-12T07:36:48.229Z	INFO	[beat]	instance/beat.go:992	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-03-12T05:12:51Z","containerized":false,"name":"ip-172-31-33-55","ip":["127.0.0.1/8","::1/128","172.31.33.55/20","fe80::a:6aff:fea2:7078/64"],"kernel_version":"4.4.0-1122-aws","mac":["02:0a:6a:a2:70:78"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"16.04.7 LTS (Xenial Xerus)","major":16,"minor":4,"patch":7,"codename":"xenial"},"timezone":"UTC","timezone_offset_sec":0,"id":"1ce62cb7754341cbbec8a0bf54a731bb"}}}
2021-03-12T07:36:48.229Z	INFO	[beat]	instance/beat.go:1021	Process info	{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/root", "exe": "/usr/share/metricbeat/bin/metricbeat", "name": "metricbeat", "pid": 2409, "ppid": 2408, "seccomp": {"mode":"filter"}, "start_time": "2021-03-12T07:36:47.379Z"}}}
2021-03-12T07:36:48.230Z	INFO	instance/beat.go:299	Setup Beat: metricbeat; Version: 7.9.1
2021-03-12T07:36:48.230Z	INFO	[index-management]	idxmgmt/std.go:184	Set output.elasticsearch.index to 'metricbeat-7.9.1' as ILM is enabled.
2021-03-12T07:36:48.232Z	INFO	eslegclient/connection.go:99	elasticsearch url: https://65.0.82.214:9200
2021-03-12T07:36:48.232Z	WARN	[tls]	tlscommon/tls_config.go:83	SSL/TLS verifications disabled.
2021-03-12T07:36:48.233Z	INFO	[publisher]	pipeline/module.go:113	Beat name: ip-172-31-33-55
2021-03-12T07:36:48.247Z	INFO	instance/beat.go:450	metricbeat start running.
2021-03-12T07:36:48.251Z	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2021-03-12T07:36:49.080Z	INFO	[monitoring]	log/log.go:153	Total non-zero metrics	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":10,"time":{"ms":16}},"total":{"ticks":150,"time":{"ms":160},"value":0},"user":{"ticks":140,"time":{"ms":144}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":9},"info":{"ephemeral_id":"b649aa5e-100c-4516-b264-f91e491ce021","uptime":{"ms":909}},"memstats":{"gc_next":21903952,"memory_alloc":16880272,"memory_total":39375568,"rss":82509824},"runtime":{"goroutines":19}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"system":{"cpu":{"cores":1},"load":{"1":0.08,"15":0.01,"5":0.02,"norm":{"1":0.08,"15":0.01,"5":0.02}}}}}}
2021-03-12T07:36:49.081Z	INFO	[monitoring]	log/log.go:154	Uptime: 910.492701ms
2021-03-12T07:36:49.081Z	INFO	[monitoring]	log/log.go:131	Stopping metrics logging.
2021-03-12T07:36:49.081Z	INFO	runtime/panic.go:975	metricbeat stopped.
2021-03-12T07:36:49.082Z	FATAL	[metricbeat]	instance/beat.go:164	Failed due to panic.	{"panic": "runtime error: index out of range [0] with length 0", "stack": "github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1.1\n\t/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:165\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:969\nruntime.goPanicIndex\n\t/usr/local/go/src/runtime/panic.go:88\ngithub.com/elastic/beats/v7/x-pack/metricbeat/module/aws.NewMetricSet\n\t/go/src/github.com/elastic/beats/x-pack/metricbeat/module/aws/aws.go:122\ngithub.com/elastic/beats/v7/x-pack/metricbeat/module/aws/cloudwatch.New\n\t/go/src/github.com/elastic/beats/x-pack/metricbeat/module/aws/cloudwatch/cloudwatch.go:113\ngithub.com/elastic/beats/v7/metricbeat/mb.(*LightMetricSet).Registration.func1\n\t/go/src/github.com/elastic/beats/metricbeat/mb/lightmetricset.go:96\ngithub.com/elastic/beats/v7/metricbeat/mb.initMetricSets\n\t/go/src/github.com/elastic/beats/metricbeat/mb/builders.go:139\ngithub.com/elastic/beats/v7/metricbeat/mb.NewModule\n\t/go/src/github.com/elastic/beats/metricbeat/mb/builders.go:65\ngithub.com/elastic/beats/v7/metricbeat/mb/module.NewWrapper\n\t/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:83\ngithub.com/elastic/beats/v7/metricbeat/mb/module.(*Factory).CheckConfig\n\t/go/src/github.com/elastic/beats/metricbeat/mb/module/factory.go:78\ngithub.com/elastic/beats/v7/libbeat/cfgfile.(*Reloader).Check\n\t/go/src/github.com/elastic/beats/libbeat/cfgfile/reload.go:155\ngithub.com/elastic/beats/v7/metricbeat/beater.(*Metricbeat).Run\n\t/go/src/github.com/elastic/beats/metricbeat/beater/metricbeat.go:234\ngithub.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch\n\t/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:456\ngithub.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1\n\t/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:189\ngithub.com/elastic/beats/v7/libbeat/cmd/instance.Run\n\t/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:190\ngithub.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1\n\t/go/src/github.com/elastic/beats/libbeat/cmd/run.go:36\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:766\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:852\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:800\nmain.main\n\t/go/src/github.com/elastic/beats/x-pack/metricbeat/main.go:21\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"}

aws.yml file
# Module: aws
# Docs: AWS module | Metricbeat Reference [7.9] | Elastic

    - module: aws
      period: 1m
      access_key_id: "access_key"
      secret_access_key: "secret_key"
      metricsets:
        - elb
        - natgateway
        - rds
        - transitgateway
        - usage
        - vpn
    - module: aws
      period: 5m
      access_key_id: "access_key"
      secret_access_key: "secret_key"
      metricsets:
        - cloudwatch
      metrics:
        - namespace: AWS/EC2
          #name: ["CPUUtilization", "DiskWriteOps"]
          resource_type: ec2:instance
          #dimensions:
          #  - name: InstanceId
          #    value: i-0686946e22cf9494a
          #statistic: ["Average", "Maximum"]
    - module: aws
      period: 5m
      access_key_id: "access_key"
      secret_access_key: "secret_key"
      metricsets:
        - dynamodb
        - ebs
        - ec2
        - lambda
        - rds
        - sns
        - sqs
    - module: aws
      period: 12h
      access_key_id: "access_key"
      secret_access_key: "secret_key"
      metricsets:
        - billing
      regions:
        - ap-south-1
    - module: aws
      period: 24h
      access_key_id: "access_key"
      secret_access_key: "secret_key"
      metricsets:
        - s3_daily_storage
        - s3_request

The permissions AWS keys having is as following
Selection_129

2

@Kaiyan_Sheng I saw your thread in Github regarding the same issue. Tagging you if you could help :slightly_smiling_face:

3

Hi :wave: Thanks for posting here!! Could you send me the github issue link please? TIA!!

Also could you try with a simple config first to see if the error still there? For example:

    - module: aws
      period: 5m
      access_key_id: "access_key"
      secret_access_key: "secret_key"
      metricsets:
        - ec2
4

Thanks @Kaiyan_Sheng for quick response.
here is the Github link. Also, I tried with simple config you mentioned above, still getting the same error.

5

Can anyone help here?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.