Filebeat error

I am using nginx as a frontend for Kibana. I am trying to load the logs onto kibana using a Filebeat module.

This is how Kibana portion of filebeat.yml file looks like

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "elasticstacktesting1.cacr.camp:80"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

I get the following error when I run the filebeat setup command:
sudo filebeat setup -e

2020-07-31T14:52:58.308-0400    INFO    instance/beat.go:647    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2020-07-31T14:52:58.308-0400    INFO    instance/beat.go:655    Beat ID: 7287adbb-ca50-4e74-8867-4a18ef2483fb
2020-07-31T14:52:58.353-0400    INFO    [beat]  instance/beat.go:983    Beat info       {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "7287adbb-ca50-4e74-8867-4a18ef2483fb"}}}
2020-07-31T14:52:58.353-0400    INFO    [beat]  instance/beat.go:992    Build info      {"system_info": {"build": {"commit": "94f7632be5d56a7928595da79f4b829ffe123744", "libbeat": "7.8.1", "time": "2020-07-21T15:12:45.000Z", "version": "7.8.1"}}}
2020-07-31T14:52:58.353-0400    INFO    [beat]  instance/beat.go:995    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.13.10"}}}
2020-07-31T14:52:58.355-0400    INFO    [beat]  instance/beat.go:999    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-07-16T17:06:31-04:00","containerized":false,"name":"elasticstacktesting1.cacr.camp","ip":["127.0.0.1/8","::1/128","172.28.152.17/24","fe80::f816:3eff:feab:32e2/64","172.17.0.1/16"],"kernel_version":"3.10.0-1127.13.1.el7.x86_64","mac":["fa:16:3e:ab:32:e2","02:42:31:b9:8e:71"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":8,"patch":2003,"codename":"Core"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"f1a3b8ee445eb30bc8a60b0d21580ef5"}}}
2020-07-31T14:52:58.355-0400    INFO    [beat]  instance/beat.go:1028   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 8080, "ppid": 8079, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2020-07-31T14:52:57.250-0400"}}}
2020-07-31T14:52:58.355-0400    INFO    instance/beat.go:310    Setup Beat: filebeat; Version: 7.8.1
2020-07-31T14:52:58.356-0400    INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'filebeat-7.8.1' as ILM is enabled.
2020-07-31T14:52:58.356-0400    INFO    eslegclient/connection.go:99    elasticsearch url: http://localhost:9200
2020-07-31T14:52:58.357-0400    INFO    [publisher]     pipeline/module.go:113  Beat name: elasticstacktesting1.cacr.camp
2020-07-31T14:52:58.359-0400    INFO    eslegclient/connection.go:99    elasticsearch url: http://localhost:9200
2020-07-31T14:52:58.366-0400    INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.8.0
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite:true` for enabling.

2020-07-31T14:52:58.398-0400    INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
2020-07-31T14:52:58.400-0400    INFO    [index-management.ilm]  ilm/std.go:139  do not generate ilm policy: exists=true, overwrite=false
2020-07-31T14:52:58.400-0400    INFO    [index-management]      idxmgmt/std.go:274      ILM policy successfully loaded.
2020-07-31T14:52:58.400-0400    INFO    [index-management]      idxmgmt/std.go:407      Set setup.template.name to '{filebeat-7.8.1 {now/d}-000001}' as ILM is enabled.
2020-07-31T14:52:58.401-0400    INFO    [index-management]      idxmgmt/std.go:412      Set setup.template.pattern to 'filebeat-7.8.1-*' as ILM is enabled.
2020-07-31T14:52:58.401-0400    INFO    [index-management]      idxmgmt/std.go:446      Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.8.1 {now/d}-000001} as ILM is enabled.
2020-07-31T14:52:58.401-0400    INFO    [index-management]      idxmgmt/std.go:450      Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-07-31T14:52:58.403-0400    INFO    template/load.go:169    Existing template will be overwritten, as overwrite is enabled.
2020-07-31T14:52:58.823-0400    INFO    template/load.go:109    Try loading template filebeat-7.8.1 to Elasticsearch
2020-07-31T14:52:59.107-0400    INFO    template/load.go:101    template with name 'filebeat-7.8.1' loaded.
2020-07-31T14:52:59.107-0400    INFO    [index-management]      idxmgmt/std.go:298      Loaded index template.
2020-07-31T14:52:59.108-0400    INFO    [index-management]      idxmgmt/std.go:309      Write alias successfully generated.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
2020-07-31T14:52:59.108-0400    INFO    kibana/client.go:118    Kibana url: http://elasticstacktesting1.cacr.camp:80
2020-07-31T14:52:59.739-0400    INFO    kibana/client.go:118    Kibana url: http://elasticstacktesting1.cacr.camp:80
2020-07-31T14:53:00.376-0400    INFO    [add_cloud_metadata]    add_cloud_metadata/add_cloud_metadata.go:93     add_cloud_metadata: hosting provider type detected as openstack, metadata={"availability_zone":"zone-r3","instance":{"id":"i-000392c2","name":"centos-7-7-7-development-gui.novalocal"},"machine":{"type":"m1.quad"},"provider":"openstack"}
2020-07-31T14:53:05.615-0400    ERROR   instance/beat.go:958    Exiting: Failed to import dashboard: Failed to load directory /usr/share/filebeat/kibana/7/dashboard:
  error loading /usr/share/filebeat/kibana/7/dashboard/749203a0-67b1-11ea-a76f-bf44814e437d.json: returned 404 to import file: <nil>. Response: {"statusCode":404,"error":"Not Found","message":"Not Found"}
  error loading /usr/share/filebeat/kibana/7/dashboard/Coredns-Overview-Dashboard.json: returned 404 to import file: <nil>. Response: {"statusCode":404,"error":"Not Found","message":"Not Found"}
  error loading /usr/share/filebeat/kibana/7/dashboard/Filebeat-Cisco-ASA.json: returned 404 to import file: <nil>. Response: {"statusCode":404,"error":"Not Found","message":"Not Found"}
  error loading /usr/share/filebeat/kibana/7/dashboard/Filebeat-Envoyproxy-Overview.json: returned 404 to import file: <nil>. Response: {"statusCode":404,"error":"Not Found","message":"Not Found"}
  error loading /usr/share/filebeat/kibana/7/dashboard/Filebeat-IBMMQ-Overview.json: returned 404 to import file: <nil>. Response: {"statusCode":404,"error":"Not Found","message":"Not Found"}
  error loading /usr/share/filebeat/kibana/7/dashboard/Filebeat-Iptables-Overview.json: returned 404 to import file: <nil>. Response: {"statusCode":404,"error":"Not Found","message":"Not Found"}
  error loading /usr/share/filebeat/kibana/7/dashboard/Filebeat-Iptables-Ubiquiti-Firewall-Overview.json: returned 404 to import file: <nil>. Response: {"statusCode":404,"error":"Not Found","message":"Not Found"}
 

Error message is little longer that this but I am getting the same 404 error.
Can someone help?

I understand that you want to load the filebeat dashboards into Kibana. In this case you have to point directly to your Kibana host on port 5601. I am assuming you have not change Kibana default port. So change this line host: "elasticstacktesting1.cacr.camp:80" to your Kibana host.

This is the error I get when I change that line to host: "elasticstacktesting1.cacr.camp:5601".

2020-07-31T17:21:16.359-0400    INFO    instance/beat.go:647    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2020-07-31T17:21:16.359-0400    INFO    instance/beat.go:655    Beat ID: 7287adbb-ca50-4e74-8867-4a18ef2483fb
2020-07-31T17:21:16.384-0400    INFO    [beat]  instance/beat.go:983    Beat info       {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "7287adbb-ca50-4e74-8867-4a18ef2483fb"}}}
2020-07-31T17:21:16.384-0400    INFO    [beat]  instance/beat.go:992    Build info      {"system_info": {"build": {"commit": "94f7632be5d56a7928595da79f4b829ffe123744", "libbeat": "7.8.1", "time": "2020-07-21T15:12:45.000Z", "version": "7.8.1"}}}
2020-07-31T17:21:16.384-0400    INFO    [beat]  instance/beat.go:995    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.13.10"}}}
2020-07-31T17:21:16.385-0400    INFO    [beat]  instance/beat.go:999    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-07-16T17:06:31-04:00","containerized":false,"name":"elasticstacktesting1.cacr.camp","ip":["127.0.0.1/8","::1/128","172.28.152.17/24","fe80::f816:3eff:feab:32e2/64","172.17.0.1/16"],"kernel_version":"3.10.0-1127.13.1.el7.x86_64","mac":["fa:16:3e:ab:32:e2","02:42:31:b9:8e:71"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":8,"patch":2003,"codename":"Core"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"f1a3b8ee445eb30bc8a60b0d21580ef5"}}}
2020-07-31T17:21:16.385-0400    INFO    [beat]  instance/beat.go:1028   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 7312, "ppid": 7311, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2020-07-31T17:21:15.280-0400"}}}
2020-07-31T17:21:16.386-0400    INFO    instance/beat.go:310    Setup Beat: filebeat; Version: 7.8.1
2020-07-31T17:21:16.386-0400    INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'filebeat-7.8.1' as ILM is enabled.
2020-07-31T17:21:16.386-0400    INFO    eslegclient/connection.go:99    elasticsearch url: http://localhost:9200
2020-07-31T17:21:16.387-0400    INFO    [publisher]     pipeline/module.go:113  Beat name: elasticstacktesting1.cacr.camp
2020-07-31T17:21:16.388-0400    INFO    eslegclient/connection.go:99    elasticsearch url: http://localhost:9200
2020-07-31T17:21:16.391-0400    INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.8.0
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite:true` for enabling.

2020-07-31T17:21:16.414-0400    INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
2020-07-31T17:21:16.415-0400    INFO    [index-management.ilm]  ilm/std.go:139  do not generate ilm policy: exists=true, overwrite=false
2020-07-31T17:21:16.415-0400    INFO    [index-management]      idxmgmt/std.go:274      ILM policy successfully loaded.
2020-07-31T17:21:16.415-0400    INFO    [index-management]      idxmgmt/std.go:407      Set setup.template.name to '{filebeat-7.8.1 {now/d}-000001}' as ILM is enabled.
2020-07-31T17:21:16.415-0400    INFO    [index-management]      idxmgmt/std.go:412      Set setup.template.pattern to 'filebeat-7.8.1-*' as ILM is enabled.
2020-07-31T17:21:16.415-0400    INFO    [index-management]      idxmgmt/std.go:446      Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.8.1 {now/d}-000001} as ILM is enabled.
2020-07-31T17:21:16.415-0400    INFO    [index-management]      idxmgmt/std.go:450      Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-07-31T17:21:16.419-0400    INFO    template/load.go:169    Existing template will be overwritten, as overwrite is enabled.
2020-07-31T17:21:16.827-0400    INFO    template/load.go:109    Try loading template filebeat-7.8.1 to Elasticsearch
2020-07-31T17:21:17.049-0400    INFO    template/load.go:101    template with name 'filebeat-7.8.1' loaded.
2020-07-31T17:21:17.050-0400    INFO    [index-management]      idxmgmt/std.go:298      Loaded index template.
2020-07-31T17:21:17.050-0400    INFO    [index-management]      idxmgmt/std.go:309      Write alias successfully generated.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
2020-07-31T17:21:17.051-0400    INFO    kibana/client.go:118    Kibana url: http://elasticstacktesting1.cacr.camp:5601
2020-07-31T17:21:17.052-0400    ERROR   instance/beat.go:958    Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://elasticstacktesting1.cacr.camp:5601/api/status fails: fail to execute the HTTP GET request: Get http://elasticstacktesting1.cacr.camp:5601/api/status: dial tcp 149.165.168.107:5601: connect: connection refused. Response: .
Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://elasticstacktesting1.cacr.camp:5601/api/status fails: fail to execute the HTTP GET request: Get http://elasticstacktesting1.cacr.camp:5601/api/status: dial tcp 149.165.168.107:5601: connect: connection refused. Response: .

Kibana is not reachable when I try "elasticstacktesting1.cacr.camp:5601" on the web browser either but it sure is reachable on "elasticstacktesting1.cacr.camp:80" or "elasticstacktesting1.cacr.camp:443". I have also enabled Let's encrypt certificate for my Kibana instance.
I am not sure what I am doing wrong.

In your first post the error message says Failed to load directory /usr/share/filebeat/kibana/7/dashboard:. Try to locate the dashboard directory on the server where you have installed Filebeat and then set the setup.dashboards.directory to point to that directory in the setup dashboards section of your filebeat.yml

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.