Filebeat Dashboard Setup is a Hassle!

tomx1 · August 25, 2021, 9:42am

I'm frustrated!!! It's such a hassle to load sample dashboards dashboards in Kibana, this should be an easy straight forward task. But somehow you guys managed to make it an hole day project. Sorry, but I'm pissed currently! I just want to load netflow sample dashboard... I'm already pushing netflow data in my cluster via filebeat and ran:

root@server:/etc/filebeat# filebeat setup --dashboards
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards

But no dashboards appear at all in my Kibana. I wish I could just download those f*ckin Dashboard / Visualiziation set somewhere and import it in Kibana and this task is done. But no, I have to spend hours for this stupid task just because someone thought "lets built this shit complicated"... grrrr

warkolm · August 25, 2021, 9:51am

What version of things are you running?

What do you see when you open Kibana and head to the Dashboards section?

tomx1 · August 25, 2021, 9:54am

The hole Elastic Stack is on 7.13.2 currently

warkolm · August 25, 2021, 9:55am

Ok thanks. And what about this part?

tomx1 · August 25, 2021, 9:57am

Only the Dashboards already existed but no netflow dashboards at all and also no netflow visualizations.

warkolm · August 25, 2021, 9:59am

Do you have the module enabled in Filebeat as well?

tomx1 · August 25, 2021, 10:02am

Sure, and it's importing the data into my filebeat index / cluster as expected:

root@server:/usr/share/logstash/modules/netflow/configuration/kibana/visualization# ll /etc/filebeat/modules.d/ |grep -v disabled
total 292K
-rw-r--r-- 1 root root  243 Jun 30 14:37 haproxy.yml
-rw-r--r-- 1 root root  645 Aug 25 08:50 netflow.yml
root@server:/usr/share/logstash/modules/netflow/configuration/kibana/visualization#
root@server:/usr/share/logstash/modules/netflow/configuration/kibana/visualization#
root@server:/usr/share/logstash/modules/netflow/configuration/kibana/visualization# cat /etc/filebeat/modules.d/netflow.yml
# Module: netflow
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.13/filebeat-module-netflow.html

- module: netflow
  log:
    enabled: true
    var:
      netflow_host: 0.0.0.0
      netflow_port: 2055
      queue_size: 32768
      detect_sequence_reset: true
      tags: ["netflow"]
      # internal_networks specifies which networks are considered internal or private
      # you can specify either a CIDR block or any of the special named ranges listed
      # at: https://www.elastic.co/guide/en/beats/filebeat/current/defining-processors.html#condition-network
      internal_networks:
        - private

warkolm · August 25, 2021, 10:04am

Can you try running setup with debug enabled and posting the output please? Debug | Filebeat Reference [7.13] | Elastic

You are correct in saying that it should be pretty simple, that command is all you should need, so something is weird up.

tomx1 · August 25, 2021, 10:06am

Here we go:

root@server:/usr/share/logstash/modules/netflow/configuration/kibana/visualization# filebeat -e setup --dashboards
2021-08-25T10:05:14.018Z        INFO    instance/beat.go:665    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2021-08-25T10:05:14.019Z        INFO    instance/beat.go:673    Beat ID: baaeefb8-4d83-4e3e-bea9-0033e53deafb
2021-08-25T10:05:14.020Z        INFO    [beat]  instance/beat.go:1014   Beat info       {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "baaeefb8-4d83-4e3e-bea9-0033e53deafb"}}}
2021-08-25T10:05:14.020Z        INFO    [beat]  instance/beat.go:1023   Build info      {"system_info": {"build": {"commit": "686ba416a74193f2e69dcfa2eb142f4364a79307", "libbeat": "7.13.2", "time": "2021-06-10T21:04:13.000Z", "version": "7.13.2"}}}
2021-08-25T10:05:14.020Z        INFO    [beat]  instance/beat.go:1026   Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":32,"version":"go1.15.13"}}}
2021-08-25T10:05:14.023Z        INFO    [beat]  instance/beat.go:1030   Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-06-21T10:54:29Z","containerized":false,"name":"server","ip":["127.0.0.1/8","::1/128","10.10.7.111/24","fe80::d43d:baff:fe86:eff6/64"],"kernel_version":"4.19.0-17-amd64","mac":["d6:3d:ba:86:ef:f6"],"os":{"type":"linux","family":"debian","platform":"debian","name":"Debian GNU/Linux","version":"10 (buster)","major":10,"minor":0,"patch":0,"codename":"buster"},"timezone":"UTC","timezone_offset_sec":0,"id":"1a53c4496a8347b4b1fad0c019bc4783"}}}
2021-08-25T10:05:14.024Z        INFO    [beat]  instance/beat.go:1059   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/usr/share/logstash/modules/netflow/configuration/kibana/visualization", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 12284, "ppid": 11928, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2021-08-25T10:05:12.830Z"}}}
2021-08-25T10:05:14.025Z        INFO    instance/beat.go:309    Setup Beat: filebeat; Version: 7.13.2
2021-08-25T10:05:14.026Z        WARN    [cfgwarn]       tlscommon/config.go:105 DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0
2021-08-25T10:05:14.026Z        INFO    eslegclient/connection.go:99    elasticsearch url: https://elastic01.fqnd.com:9200
2021-08-25T10:05:14.027Z        WARN    [tls]   tlscommon/tls_config.go:98      SSL/TLS verifications disabled.
2021-08-25T10:05:14.027Z        INFO    eslegclient/connection.go:99    elasticsearch url: https://elastic02.fqnd.com:9200
2021-08-25T10:05:14.027Z        WARN    [tls]   tlscommon/tls_config.go:98      SSL/TLS verifications disabled.
2021-08-25T10:05:14.027Z        INFO    eslegclient/connection.go:99    elasticsearch url: https://elastic03.fqnd.com:9200
2021-08-25T10:05:14.027Z        WARN    [tls]   tlscommon/tls_config.go:98      SSL/TLS verifications disabled.
2021-08-25T10:05:14.027Z        INFO    eslegclient/connection.go:99    elasticsearch url: https://elastic04.fqnd.com:9200
2021-08-25T10:05:14.027Z        WARN    [tls]   tlscommon/tls_config.go:98      SSL/TLS verifications disabled.
2021-08-25T10:05:14.028Z        INFO    eslegclient/connection.go:99    elasticsearch url: https://elastic05.fqnd.com:9200
2021-08-25T10:05:14.028Z        WARN    [tls]   tlscommon/tls_config.go:98      SSL/TLS verifications disabled.
2021-08-25T10:05:14.029Z        INFO    [publisher]     pipeline/module.go:113  Beat name: server
Loading dashboards (Kibana must be running and reachable)
2021-08-25T10:05:14.033Z        INFO    kibana/client.go:119    Kibana url: https://logging.fqnd.com:443
2021-08-25T10:05:17.722Z        INFO    kibana/client.go:119    Kibana url: https://logging.fqnd.com:443
2021-08-25T10:05:19.620Z        INFO    instance/beat.go:848    Kibana dashboards successfully loaded.
Loaded dashboards

warkolm · August 25, 2021, 10:09am

Thanks for sticking with us here!

And what do you see if you go Stack Management > Saved Objects in Kibana? Is there any reference to Netflow there? Otherwise, what can you see if you filter on Type: dashboard?

tomx1 · August 25, 2021, 10:12am

0 results for netflow visualiziations and 5 results for dashboards (which I had already before, so nothing changed here). It just came into my mind that I already opened an issue on github for this two months ago: HaProxy Sample Dashboards not loading by Filebeat · Issue #26635 · elastic/beats · GitHub

@warkolm Is it somehow possible to download the dashboard / visualization set somewhere just to get it done for the moment?

warkolm · August 25, 2021, 10:14am

I am not 100% sure sorry. Let me move this to the Beats category though, someone there might have a better idea of what's happening.

tomx1 · August 25, 2021, 2:34pm

Just updated filebeat to 7.14.0... no change. Man, thats really frustrating as I don't even know if those dashboards and visualizations are what I am looking for. So I don't even know if it's worth the work I'm doing right now.
Guys, it would be really helpful, if you upload those dashboards and visualizations somewhere so the useres can download and test them in Kibana without having to fight with any kind of setup! Just download, install, test > done

I also just tried to import the netflow files stored in /usr/share/filebeat/kibana/7/dashboard into Kibana. But Kibana don't likes those files.

gsmitt · August 27, 2021, 1:36am

Hello,
I had a problem like this and I also feel your frustration. Not sure if it will work but I solved my similar issue by changing my Tenant.

Hope that helps

stephenb · August 27, 2021, 1:45am

One thing perhaps

You need to run setup without the

--dashboard

Run

filebeat -e setup

When you add the --dashboard flag that's all it loads It does not load the index templates and mappings and ingest pipeline which are needed in order correctly load and parse the data.

This is why in the quick start it just says to run setup see here

Also I see Logstash may be involved are you passing the data through Logstash? If so you need a proper Logstash config that support the modules.

I have a sample here with some discussion

################################################
# beats->logstash->es default config.
################################################
input {
  beats {
    port => 5044
  }
}

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "http://localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      pipeline => "%{[@metadata][pipeline]}" 
      user => "elastic"
      password => "secret"
    }
  } else {
    elasticsearch {
      hosts => "http://localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      user => "elastic"
      password => "secret"
    }
  }
}

Also I am bit confused of whether you're trying to use the filebeat module or the log stash module you should use one of the other not both.

tomx1 · September 15, 2021, 9:32am

Sorry for my late response. Thanks guys for your help, it turns out that at the end it was a miserable user error. As we are using spaces in Kibana and filebeat is loading dashboards bei default into the default space, I did'nt saw the dashboards and visualizations.
Changing the setting to the desired space via 'space.id:' setting solved the issue.

However, before I did that I renamed all the dashboards in /usr/share/filebeat/kibana/7/dashboard/ to *.disabled (except netflow) as it seems otherwise all dashboards and visualizations are loaded into kibana which ends up with a mess.

warkolm · September 15, 2021, 9:11pm

Glad you got it all sorted out!

system · October 13, 2021, 9:11pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Blank dashboards after setup Kibana	7	1659	September 3, 2019
Importing Kibana dashboards: fail to create the Kibana loader Beats filebeat	3	5303	March 7, 2018
Issues with Index Pattern Search - Kibana/Filebeat Pre Loaded Dashboards Beats filebeat	2	638	June 23, 2020
Filebeat, nginx module dashboards are broken after migrate to 6.7 Beats filebeat	5	412	May 28, 2019
No sample dashboard or Visualize in new installed Kibana Kibana	3	1636	January 28, 2018

Filebeat Dashboard Setup is a Hassle!

Related topics