Panic: runtime error: index out of range goroutine 1 [running]: while running metricbeat setup

prat · November 21, 2020, 11:58am

running metricbeat setup command through ansible but getting below error. Can someone please sugesst where to check to fix this issue. After 2nd playbook run can see this host is skipped or ignored (may be due to when condition) but issues still exists if run this command manually on server.

TASK [metricbeat : Metricbeat default dashboards for Kibana] ***********************************************************************************************************
fatal: [server]: FAILED! => {
    "changed": true,
    "cmd": "metricbeat setup -e -E output.kafka.enabled=false -E output.elasticsearch.hosts=['http://<IP of server1>:5200', 'http://<IP of server2>:5200', 'http://<IP of server3>:5200'] -E output.elasticsearch.username=elastic -E output.elasticsearch.password=<Password> -E setup.kibana.host=<IP of server1> > /var/log/metricbeat_dashboard_load.log",
    "delta": "0:00:00.122571",
    "end": "2020-11-20 15:21:45.747724",
    "rc": 2,
    "start": "2020-11-20 15:21:45.625153"
}

STDERR:

panic: runtime error: index out of range

goroutine 1 [running]:
github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/internal/parse.(*flagParser).parseArray(0xc0004dd830, 0x3f2f4e0, 0xc0004dd758, 0x1de37a8, 0x7ffd0000002c)
        /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/internal/parse/parse.go:125 +0x3c3
github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/internal/parse.(*flagParser).parseValue(0xc0004dd830, 0x4517e63, 0x1, 0x4654878, 0x0, 0x1b, 0x1)
        /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/internal/parse/parse.go:102 +0x1e7
github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/internal/parse.(*flagParser).parse(0xc0004dd830, 0x1c, 0x4654878, 0x7ffdd60d5da2, 0x1c)
        /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/internal/parse/parse.go:67 +0x6d
github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/internal/parse.Value(0x7ffdd60d5da2, 0x1c, 0x4517e72, 0x1, 0x0, 0x1)
        /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/internal/parse/parse.go:56 +0x84
github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/flag.NewFlagKeyValue.func1(0x7ffdd60d5d87, 0x37, 0x0, 0x1d06a0f, 0xc0004dd9c0, 0xc87d504f, 0x84177415c81ea561)
        /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/flag/value.go:69 +0x325
github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/flag.(*FlagValue).Set(0xc0001400f0, 0x7ffdd60d5d87, 0x37, 0x4517e7a, 0xc0001b80d0)
        /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/flag/util.go:64 +0x42
github.com/elastic/beats/libbeat/common.(*SettingsFlag).Set(0xc0001400f0, 0x7ffdd60d5d87, 0x37, 0x1, 0xc0001b8160)
        /go/src/github.com/elastic/beats/libbeat/common/flags.go:209 +0x40
github.com/elastic/beats/vendor/github.com/spf13/pflag.(*FlagSet).Set(0xc0000bd0e0, 0x4517e7a, 0x1, 0x7ffdd60d5d87, 0x37, 0x6ef7700, 0xc00013aa80)
        /go/src/github.com/elastic/beats/vendor/github.com/spf13/pflag/flag.go:432 +0xe6
github.com/elastic/beats/vendor/github.com/spf13/pflag.(*FlagSet).Parse.func1(0xc0000b9ea0, 0x7ffdd60d5d87, 0x37, 0xc00013aa90, 0x1)
        /go/src/github.com/elastic/beats/vendor/github.com/spf13/pflag/flag.go:1028 +0x54
github.com/elastic/beats/vendor/github.com/spf13/pflag.(*FlagSet).parseSingleShortArg(0xc0000bd0e0, 0x7ffdd60d5d85, 0x1, 0xc0000f20c0, 0x9, 0xa, 0xc00034c6d0, 0x7ffdd60d5d67, 0x0, 0xc0000f20b0, ...)
        /go/src/github.com/elastic/beats/vendor/github.com/spf13/pflag/flag.go:964 +0x298
github.com/elastic/beats/vendor/github.com/spf13/pflag.(*FlagSet).parseShortArg(0xc0000bd0e0, 0x7ffdd60d5d84, 0x2, 0xc0000f20c0, 0x9, 0xa, 0xc00034c6d0, 0xc0000f20b0, 0xa, 0xb, ...)
        /go/src/github.com/elastic/beats/vendor/github.com/spf13/pflag/flag.go:974 +0xd4
github.com/elastic/beats/vendor/github.com/spf13/pflag.(*FlagSet).parseArgs(0xc0000bd0e0, 0xc0000f20c0, 0x9, 0xa, 0xc00034c6d0, 0xc0000bd770, 0xc0000bd0e0)
        /go/src/github.com/elastic/beats/vendor/github.com/spf13/pflag/flag.go:1005 +0x6a6
github.com/elastic/beats/vendor/github.com/spf13/pflag.(*FlagSet).Parse(0xc0000bd0e0, 0xc0000f2000, 0x15, 0x16, 0x0, 0xc000052c20)
        /go/src/github.com/elastic/beats/vendor/github.com/spf13/pflag/flag.go:1031 +0xe3
github.com/elastic/beats/vendor/github.com/spf13/cobra.(*Command).ParseFlags(0xc0001ce240, 0xc0000f2000, 0x15, 0x16, 0x6f8ec20, 0xc0004dde98)
        /go/src/github.com/elastic/beats/vendor/github.com/spf13/cobra/command.go:1373 +0x9c
github.com/elastic/beats/vendor/github.com/spf13/cobra.(*Command).execute(0xc0001ce240, 0xc0000f2000, 0x15, 0x16, 0xc0001ce240, 0xc0000f2000)
        /go/src/github.com/elastic/beats/vendor/github.com/spf13/cobra/command.go:647 +0x8f
github.com/elastic/beats/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc0001c6000, 0x3b342ee, 0xc0004ddf88, 0xc0000560b8)
        /go/src/github.com/elastic/beats/vendor/github.com/spf13/cobra/command.go:785 +0x2ca
github.com/elastic/beats/vendor/github.com/spf13/cobra.(*Command).Execute(...)
        /go/src/github.com/elastic/beats/vendor/github.com/spf13/cobra/command.go:738
main.main()
        /go/src/github.com/elastic/beats/x-pack/metricbeat/main.go:21 +0x30

MSG:

non-zero return code

shaunak · November 24, 2020, 2:56am

Which version of Metricbeat are you running?

Can you run the following command manually on the server and post the results here please?

metricbeat -E "output.elasticsearch.hosts=['foo', 'bar']" export config

prat · November 24, 2020, 6:43am

Hi Shaunak,

Thanks for your reply. Please find the output below.

metricbeat version

metricbeat version 7.4.0 (amd64), libbeat 7.4.0 [f940c36884d3749901a9c99bea5463a6030cdd9c built 2019-09-27 07:51:21 +0000 UTC]

metricbeat -E "output.elasticsearch.hosts=['foo','bar']" export config

Failed to initialize 'export' command: error unpacking config data: more than one namespace configured accessing 'output' (source:'/etc/metricbeat/metricbeat.yml').

cat /etc/metricbeat/metricbeat.yml

# Ansible managed

name: haserver
fields_under_root: true
fields:
  host_id: haserver

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true

setup.dashboards.enabled: true
setup.kibana:
  host: "server1:8601"
  username: elastic
  password: Password


output.kafka:
  hosts: ['server1:3997', 'server2:3997', 'server3:3997']
  topic: "metricbeat"
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  max_message_bytes: 1000000

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

server1 where elasticsearch_1, logstash_1, kibana_1, kafka_1 getting installed.
server2 where elasticsearch_2, logstash_2, kibana_2, kafka_2, getting installed.
server3 where kafka_3 is installed.

shaunak · November 24, 2020, 7:33am

Apologies, I missed that you were disabling the kafka output. Please try the following command manually and post the results.

metricbeat -E "output.kafka.enabled=false" -E "output.elasticsearch.hosts=['foo', 'bar']" export config

prat · November 24, 2020, 9:00am

Hi Shaunak,

Thanks. Please find below output.

metricbeat -E "output.kafka.enabled=false" -E "output.elasticsearch.hosts=['foo','bar']" export config

fields:
  host_id: haserver
fields_under_root: true
metricbeat:
  config:
    modules:
      path: /etc/metricbeat/modules.d/*.yml
      reload:
        enabled: true
name: haserver
output:
  elasticsearch:
    hosts:
    - foo
    - bar
  kafka:
    enabled: false
    hosts:
    - server1:3997
    - server2:3997
    - server3:3997
    max_message_bytes: 1000000
    partition:
      round_robin:
        reachable_only: false
    required_acks: 1
    topic: metricbeat
path:
  config: /etc/metricbeat
  data: /var/lib/metricbeat
  home: /usr/share/metricbeat
  logs: /var/log/metricbeat
processors:
- add_host_metadata: null
- add_cloud_metadata: null
setup:
  dashboards:
    enabled: true
  kibana:
    host: server1:8601
    password: Password
    username: elastic

shaunak · November 24, 2020, 10:25am

Thanks, this tells me that you just need to add quotes (") around your ['http://<IP of server1>:5200', 'http://<IP of server2>:5200', 'http://<IP of server3>:5200'] and it should work after that.

prat · November 24, 2020, 1:49pm

Thank you. You mean to say quotes should be around metricbeat setup command or in /etc/metricbeat/metricbeat.yml file.

i.e

metricbeat setup -e -E output.kafka.enabled=false -E output.elasticsearch.hosts='['http://<IP of server1>:5200', 'http://<IP of server2>:5200', 'http://<IP of server3>:5200']' -E output.elasticsearch.username=elastic -E output.elasticsearch.password=<Password> -E setup.kibana.host=<IP of server1> > /var/log/metricbeat_dashboard_load.log

or in /etc/metricbeat/metricbeat.yml file permanently for below hosts line.

output.kafka: 
hosts: '['http://<IP of server1>:5200', 'http://<IP of server2>:5200', 'http://<IP of server3>:5200']'

Thanks,

shaunak · November 24, 2020, 4:18pm

In the metricbeat setup command like this: ... -E output.elasticsearch.hosts="['http://<IP of server1>:5200', 'http://<IP of server2>:5200', 'http://<IP of server3>:5200']" ...

steffens · November 24, 2020, 4:43pm

There is no need for [. Actually the parse does not always require the string to be escape. This might already work:

... -E output.elasticsearch.hosts='http://<ip1>:5200,http://<ip2>:5200,http://<ip3>:5200'

ipv6 with [] can trip the parser at times and quotation\escaping will be needed.

In your config file do not use '['abc']'. The yaml parser won't know the difference between the inner and outer quotes. In general, in the YAML file you just need to add an array like:

hosts: ['http://<IP of server1>:5200', 'http://<IP of server2>:5200', 'http://<IP of server3>:5200']

or

hosts:
  - 'http://<IP of server1>:5200'
  - 'http://<IP of server2>:5200'
  - 'http://<IP of server3>:5200'

Problem with CLI and -E is often that it must go through multiple layers of unescaping and unqoting (e.g. ansible, shell, internal), which at times makes it difficult to say how the string would look like once it hits the parser. This is why [ and quotes on strings via CLI are optional.

prat · November 24, 2020, 4:46pm

Thanks. able to run command but with error.

metricbeat setup -e -E output.kafka.enabled=false -E output.elasticsearch.hosts="['http://<IP of server1>:5200', 'http://<IP of server2>:5200', 'http://<IP of server3>:5200']" -E output.elasticsearch.username=elastic -E output.elasticsearch.password=Password -E setup.kibana.host=<IP of server1> > /var/log/metricbeat_dashboard_load.log
2020-11-24T22:00:46.325+0530    INFO    instance/beat.go:607    Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat]
2020-11-24T22:00:46.326+0530    INFO    instance/beat.go:615    Beat ID: ecf3ec7d-779f-4574-a7de-cc533c7a5a91
2020-11-24T22:00:46.331+0530    INFO    [beat]  instance/beat.go:903    Beat info       {"system_info": {"beat": {"path": {"config": "/etc/metricbeat", "data": "/var/lib/metricbeat", "home": "/usr/share/metricbeat", "logs": "/var/log/metricbeat"}, "type": "metricbeat", "uuid": "ecf3ec7d-779f-4574-a7de-cc533c7a5a91"}}}
2020-11-24T22:00:46.332+0530    INFO    [beat]  instance/beat.go:912    Build info      {"system_info": {"build": {"commit": "f940c36884d3749901a9c99bea5463a6030cdd9c", "libbeat": "7.4.0", "time": "2019-09-27T07:51:21.000Z", "version": "7.4.0"}}}
2020-11-24T22:00:46.332+0530    INFO    [beat]  instance/beat.go:915    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.12.9"}}}
2020-11-24T22:00:46.335+0530    INFO    [beat]  instance/beat.go:919    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-11-17T05:56:44+05:30","containerized":false,"name":"<hostname of server>","ip":["127.0.0.1/8","<ip of haserver>/23","<ip of havserver>/32"],"kernel_version":"3.10.0-1127.18.2.el7.x86_64","mac":["00:50:56:a0:79:ad"],"os":{"family":"redhat","platform":"rhel","name":"Red Hat Enterprise Linux Server","version":"7.8 (Maipo)","major":7,"minor":8,"patch":0,"codename":"Maipo"},"timezone":"IST","timezone_offset_sec":19800,"id":"002f03b9b0cf47f2972ff63c4380fbf3"}}}
2020-11-24T22:00:46.337+0530    INFO    [beat]  instance/beat.go:948    Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/root", "exe": "/usr/share/metricbeat/bin/metricbeat", "name": "metricbeat", "pid": 65516, "ppid": 65351, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2020-11-24T22:00:46.070+0530"}}}
2020-11-24T22:00:46.337+0530    INFO    instance/beat.go:292    Setup Beat: metricbeat; Version: 7.4.0
2020-11-24T22:00:46.337+0530    INFO    [index-management]      idxmgmt/std.go:178      Set output.elasticsearch.index to 'metricbeat-7.4.0' as ILM is enabled.
2020-11-24T22:00:46.338+0530    INFO    elasticsearch/client.go:170     Elasticsearch url: http://<IP of server1>:5200
2020-11-24T22:00:46.338+0530    INFO    elasticsearch/client.go:170     Elasticsearch url: http://<IP of server2>:5200
2020-11-24T22:00:46.339+0530    INFO    elasticsearch/client.go:170     Elasticsearch url: http://<IP of server3>:5200
2020-11-24T22:00:46.339+0530    INFO    [publisher]     pipeline/module.go:97   Beat name: <hostname of haserver>
2020-11-24T22:00:46.343+0530    INFO    elasticsearch/client.go:170     Elasticsearch url: http://<IP of server1>:5200
2020-11-24T22:00:46.343+0530    INFO    elasticsearch/client.go:170     Elasticsearch url: http://<IP of server2>:5200
2020-11-24T22:00:46.343+0530    INFO    elasticsearch/client.go:170     Elasticsearch url: http://<IP of server3>:5200
2020-11-24T22:00:49.332+0530    INFO    add_cloud_metadata/add_cloud_metadata.go:87     add_cloud_metadata: hosting provider type not detected.
2020-11-24T22:02:16.343+0530    ERROR   elasticsearch/elasticsearch.go:260      Error connecting to Elasticsearch at http://<IP of server1>:5200: Get http://<IP of server1>:5200: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020-11-24T22:03:46.344+0530    ERROR   elasticsearch/elasticsearch.go:260      Error connecting to Elasticsearch at http://<IP of server2>:5200: Get http://<IP of server2>:5200: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020-11-24T22:05:16.344+0530    ERROR   elasticsearch/elasticsearch.go:260      Error connecting to Elasticsearch at http://<IP of server3>:5200: Get http://<IP of server3>:5200: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020-11-24T22:05:16.345+0530    ERROR   instance/beat.go:878    Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch http://<IP of server1>:5200: Get http://<IP of server1>:5200: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Error connection to Elasticsearch http://<IP of server2>:5200: Get http://<IP of server2>:5200: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Error connection to Elasticsearch http://<IP of server3>:5200: Get http://<IP of server3>:5200: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)]
Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch http://<IP of server1>:5200: Get http://<IP of server1>:5200: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Error connection to Elasticsearch http://<IP of server2>:5200: Get http://<IP of server2>:5200: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Error connection to Elasticsearch http://<IP of server3>:5200: Get http://<IP of server3>:5200: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)]

prat · November 24, 2020, 5:21pm

getting same output as above command.

steffens · November 25, 2020, 8:48am

The Error you get is on the HTTP level. The TCP connection was established and the HTTP request was send. But there is no HTTP response. Do you have a proxy or some other firewal rules in between the Beat and Elasticsearch?

Beats send a Get / on init in order to check if ES is available and which version is installed in order to accomodate for some version differences. This HTTP request is failing here.

system · December 23, 2020, 8:48am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.