Metricbeat: [logstash.node] node/node.go:79 could not fetch node pipelines: HTTP error 405 in : 405 Method Not Allowed

I created one-node elasticsearch(7.9.3) and one-node logstash(7.9.3). Logstash can transfer logs via https without problem. I want to use metricbeat to monitor logstash.
After I installed metricbeat in logstash node. Metricbeat can monitor linux server fine.

I used following setting and commands to enable logstash-xpack module.

 # metricbeat modules disable logstash
 # metricbeat modules enable logstash-xpack
 # cat /etc/metricbeat/modules.d/logstash-xpack.yml
 # Module: logstash
 # Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.9/metricbeat-module-logstash.html

 - module: logstash
   xpack.enabled: true
   period: 10s
   hosts: ["https://10.168.3.251:9200"]
   protocol: "https"
   username: "elastic"
   password: "changeme"
   ssl.certificate_authorities: '/etc/kibana/elasticsearch-ca.pem'

 # metricbeat -c metricbeat.yml -e

I got following errors.

 2020-10-30T18:02:43.994+0900    INFO    instance/beat.go:640    Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat]
 2020-10-30T18:02:43.994+0900    INFO    instance/beat.go:648    Beat ID: e314dabe-c8f1-4731-8b24-d571cf006752
 2020-10-30T18:02:43.999+0900    INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
 2020-10-30T18:02:43.999+0900    INFO    [beat]  instance/beat.go:976    Beat info       {"system_info": {"beat": {"path": {"config": "/etc/metricbeat", "data": "/var/lib/metricbeat", "home": "/usr/share/metricbeat", "logs": "/var/log/metricbeat"}, "type": "metricbeat", "uuid": "e314dabe-c8f1-4731-8b24-d571cf006752"}}}
 2020-10-30T18:02:43.999+0900    INFO    [beat]  instance/beat.go:985    Build info      {"system_info": {"build": {"commit": "7aab6a9659749802201db8020c4f04b74cec2169", "libbeat": "7.9.3", "time": "2020-10-16T09:21:04.000Z", "version": "7.9.3"}}}
 2020-10-30T18:02:43.999+0900    INFO    [beat]  instance/beat.go:988    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":6,"version":"go1.14.7"}}}
 2020-10-30T18:02:44.002+0900    INFO    [beat]  instance/beat.go:992    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-10-28T19:28:14+09:00","containerized":false,"name":"audit","ip":["127.0.0.1/8","::1/128","10.168.3.251/8","240d:1a:1e0:5f00:215:5dff:feb4:ed2a/64","fe80::215:5dff:feb4:ed2a/64"],"kernel_version":"3.10.0-1127.19.1.el7.x86_64","mac":["00:15:5d:b4:ed:2a"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":8,"patch":2003,"codename":"Core"},"timezone":"JST","timezone_offset_sec":32400,"id":"c0864aa9a15f472db5686a5b4bff2117"}}}
 2020-10-30T18:02:44.002+0900    INFO    [beat]  instance/beat.go:1021   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/etc/metricbeat", "exe": "/usr/share/metricbeat/bin/metricbeat", "name": "metricbeat", "pid": 18233, "ppid": 17348, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-10-30T18:02:43.150+0900"}}}
 2020-10-30T18:02:44.003+0900    INFO    instance/beat.go:299    Setup Beat: metricbeat; Version: 7.9.3
 2020-10-30T18:02:44.003+0900    INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'metricbeat-7.9.3' as ILM is enabled.
 2020-10-30T18:02:44.004+0900    INFO    eslegclient/connection.go:99    elasticsearch url: https://10.168.3.251:9200
 2020-10-30T18:02:44.004+0900    INFO    [publisher]     pipeline/module.go:113  Beat name: audit
 2020-10-30T18:02:44.036+0900    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
 2020-10-30T18:02:44.036+0900    INFO    instance/beat.go:450    metricbeat start running.
 2020-10-30T18:02:44.044+0900    WARN    [cfgwarn]       pageinfo/pageinfo.go:53 BETA: The linux pageinfo metricset is beta.
 2020-10-30T18:02:44.046+0900    INFO    filesystem/filesystem.go:57     Ignoring filesystem types: sysfs, rootfs, ramfs, bdev, proc, cgroup, cpuset, tmpfs, devtmpfs, debugfs, securityfs, sockfs, dax, bpf, pipefs, configfs, devpts, hugetlbfs, autofs, pstore, efivarfs, mqueue, selinuxfs
 2020-10-30T18:02:44.046+0900    INFO    [system.fsstat] fsstat/fsstat.go:56     Ignoring filesystem types: %ssysfs, rootfs, ramfs, bdev, proc, cgroup, cpuset, tmpfs, devtmpfs, debugfs, securityfs, sockfs, dax, bpf, pipefs, configfs, devpts, hugetlbfs, autofs, pstore, efivarfs, mqueue, selinuxfs
 2020-10-30T18:02:44.046+0900    INFO    cfgfile/reload.go:164   Config reloader started
 2020-10-30T18:02:44.049+0900    WARN    [cfgwarn]       pageinfo/pageinfo.go:53 BETA: The linux pageinfo metricset is beta.
 2020-10-30T18:02:44.053+0900    INFO    filesystem/filesystem.go:57     Ignoring filesystem types: sysfs, rootfs, ramfs, bdev, proc, cgroup, cpuset, tmpfs, devtmpfs, debugfs, securityfs, sockfs, dax, bpf, pipefs, configfs, devpts, hugetlbfs, autofs, pstore, efivarfs, mqueue, selinuxfs
 2020-10-30T18:02:44.053+0900    INFO    [system.fsstat] fsstat/fsstat.go:56     Ignoring filesystem types: %ssysfs, rootfs, ramfs, bdev, proc, cgroup, cpuset, tmpfs, devtmpfs, debugfs, securityfs, sockfs, dax, bpf, pipefs, configfs, devpts, hugetlbfs, autofs, pstore, efivarfs, mqueue, selinuxfs
 2020-10-30T18:02:44.061+0900    INFO    cfgfile/reload.go:224   Loading of config files completed.
 2020-10-30T18:02:44.112+0900    ERROR   [logstash.node] node/node.go:79 could not fetch node pipelines: HTTP error 405 in : 405 Method Not Allowed
 2020-10-30T18:02:44.112+0900    ERROR   [logstash.node_stats]   node_stats/node_stats.go:73     json: cannot unmarshal object into Go struct field .version of type string

How can I fix this 405 error? Thank you in advance.

Hi!

Is your configuration valid? I see a protocol setting which I cannot spot on https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-logstash.html or https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-ssl.html.

Also the metricset collects for /_node, can you check if it is reachable using a curl command?

Hi Mark

Thank you for your reply.
My configuration is valid.
I misunderstood the "hosts" in modules.d/logstash-xpack.yml.
It should point to logstash HTTP API not to the elasticsearch URL.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.