503 error from Kibana while running Filebeat setup

I am trying to run the command

filebeat setup -e

and this is the error I get

{"log.level":"info","@timestamp":"2023-10-05T02:55:42.666Z","log.origin":{"file.name":"instance/beat.go","file.line":783},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-05T02:55:42.667Z","log.origin":{"file.name":"instance/beat.go","file.line":791},"message":"Beat ID: bdaac60f-fd36-4833-9097-1720b7bac3a7","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-05T02:55:42.673Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1303},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/etc/filebeat","data":"/var/lib/filebeat","home":"/usr/share/filebeat","logs":"/var/log/filebeat"},"type":"filebeat","uuid":"bdaac60f-fd36-4833-9097-1720b7bac3a7"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-10-05T02:55:42.673Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1312},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"480bccf4f0423099bb2c0e672a44c54ecd7a805e","libbeat":"8.10.2","time":"2023-09-18T18:09:06.000Z","version":"8.10.2"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-10-05T02:55:42.673Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1315},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":2,"version":"go1.20.7"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-10-05T02:55:42.674Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1321},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2023-10-05T02:34:02Z","containerized":false,"name":"suricata","ip":["127.0.0.1","::1","159.223.222.118","10.18.0.6","2a03:b0c0:2:d0::e1f:c001","fe80::ecc5:d6ff:fe62:2951","10.133.0.3","fe80::bcc1:9bff:fed9:8ac6"],"kernel_version":"6.1.0-12-amd64","mac":["ee:c5:d6:62:29:51","be:c1:9b:d9:8a:c6"],"os":{"type":"linux","family":"debian","platform":"debian","name":"Debian GNU/Linux","version":"12 (bookworm)","major":12,"minor":0,"patch":0,"codename":"bookworm"},"timezone":"UTC","timezone_offset_sec":0,"id":"f160b40b6d64a29c1b4c5dea651d6916"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-10-05T02:55:42.675Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1350},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","perfmon","bpf","checkpoint_restore"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","perfmon","bpf","checkpoint_restore"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","perfmon","bpf","checkpoint_restore"],"ambient":null},"cwd":"/home/navjot","exe":"/usr/share/filebeat/bin/filebeat","name":"filebeat","pid":1196,"ppid":1195,"seccomp":{"mode":"disabled","no_new_privs":false},"start_time":"2023-10-05T02:55:42.540Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-10-05T02:55:42.675Z","log.origin":{"file.name":"instance/beat.go","file.line":329},"message":"Setup Beat: filebeat; Version: 8.10.2","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-10-05T02:55:42.677Z","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-05T02:55:42.678Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://10.133.0.2:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-05T02:55:42.678Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":105},"message":"Beat name: suricata","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-05T02:55:42.678Z","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":135},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-05T02:55:42.679Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://10.133.0.2:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-10-05T02:55:42.690Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":91},"message":"error fetching EC2 Identity Document: operation error ec2imds: GetInstanceIdentityDocument, http response error StatusCode: 404, request to EC2 IMDS failed.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-05T02:55:42.690Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":104},"message":"add_cloud_metadata: hosting provider type detected as digitalocean, metadata={\"cloud\":{\"instance\":{\"id\":\"377586531\"},\"provider\":\"digitalocean\",\"region\":\"ams3\",\"service\":{\"name\":\"Droplets\"}}}","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-05T02:55:42.710Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":179},"message":"'ca_trusted_fingerprint' set, looking for matching fingerprints","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-05T02:55:42.710Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":199},"message":"CA certificate matching 'ca_trusted_fingerprint' found, adding it to 'certificate_authorities'","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-05T02:55:42.720Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.10.2","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-05T02:55:42.720Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":230},"message":"Auto ILM enable success.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-05T02:55:42.724Z","log.logger":"index-management.ilm","log.origin":{"file.name":"ilm/std.go","file.line":128},"message":"ILM policy filebeat successfully created.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-05T02:55:42.724Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":365},"message":"Set settings.index.lifecycle.name in template to {filebeat {\"policy\":{\"phases\":{\"hot\":{\"actions\":{\"rollover\":{\"max_age\":\"30d\",\"max_primary_shard_size\":\"50gb\"}}}}}}} as ILM is enabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-05T02:55:42.739Z","log.logger":"template","log.origin":{"file.name":"template/load.go","file.line":245},"message":"Existing template will be overwritten, as overwrite is enabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-05T02:55:44.464Z","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":159},"message":"Try loading template filebeat-8.10.2 to Elasticsearch","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-05T02:55:44.565Z","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":127},"message":"Template with name \"filebeat-8.10.2\" loaded.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-05T02:55:44.569Z","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":143},"message":"Data stream with name \"filebeat-8.10.2\" already exists.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-10-05T02:55:44.569Z","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":266},"message":"Loaded index template.","service.name":"filebeat","ecs.version":"1.6.0"}
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
{"log.level":"info","@timestamp":"2023-10-05T02:55:44.570Z","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":183},"message":"Kibana url: http://10.133.0.2:5601","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-10-05T02:55:44.580Z","log.origin":{"file.name":"instance/beat.go","file.line":1278},"message":"Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://10.133.0.2:5601/api/status fails: status=503. Response: {\"status\":{\"overall\":{\"level\":\"unavailable\"}}}","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://10.133.0.2:5601/api/status fails: status=503. Response: {"status":{"overall":{"level":"unavailable"}}}

This is a brand new installation and Kibana and Elasticsearch are both hosted on the same server while Filebeat is on a different server. I can connect to Elasticsearch from the filebeat server but not Kibana. Please advise.

Hi @nspeaks,

Welcome back. Can you confirm that your Kibana instance is running, that the kibana settings in filebeat have the correct host and port and that the Kibana endpoint (host and post) is accessible from the filebeat server via curl?

If you could share your config that would be useful too.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.