Hello guys,
Good afternoon.
This is the first time I worked with the ELK platform in terms of analysing input data from different sources and generate different vizualization patterns.
I've been working with the following setup for ELK:
Cent Os 7 + Elastic 7.7 + Kibana 7.7 + Logstash 7.7 + Filebeat 7.7 - 19:42 - 23/05/2020
Net-Tools
wget
nmap
openssh
firewalld
yum update $$ upgrade
hostname
links - (Testing Http)
Telnet
p7zip
dnf
Java JDK 8.0
ElasticSearch 7.7
Kibana 7.7
Logstash 7.7
Filebeat 7.7
Nginx as Reverse Proxy
Dashboards already configured for Syslog
Collecting Filebeat Data From Fedora
I am intending to use in my company the best ELK deployment style that will fit in our infrastructure in terms of reaction and correction speed of any kind of performance and security issue. I am running several tests with both Logstash and Filebeats to collect and parse data from my VMs here, and also working with the Built-In Module Dashboards for Syslog and Netflow analysis on Kibana that Filebeat 7.7 setup automatically once you run the service.
My issue is that whenever I try to run Filebeat for setting up the Dashboards on Kibana from artifact.elastic.co, or the local Dashboard Configuration from Filebeat Installation Directory, I end up having an issue after selecting the Dashboards Vizualization Could not locate that Index-Pattern (Id: filebeat-*).
The issue is showing on all of them:
Netflow Overview
Netflow Conversation Partners
Netflow Traffic Analysis
Netflow Top N
Netflow Geo Location
Netflow Autonomous Systems
Netflow Flow Exporters
Netflow Raw Flow Records
OBS: The same issue happens with other Pre Built Dashboards. For Syslog Module my configuration works perfectly.
Here are my goals for filebeat now:
1 - Setup a different Index-Pattern for all Pre-Loaded or Built-In Kibana Dashboards and not the "filebeat-*". I was not able to achieve because only few Pre Loaded Dashboards accepted setup.dashboards.index: "normal-*" and the one I needed the most which was the Netflow Module did not change anything and kept trying to read the filebeat-*
2 - Having the same Index Pattern on Kibana Dashboards and Elasticsearch for Syslog Module. (I already achieved this Goal)
3 - Create different Index Pattern Outputs to Elasticsearch for logs I think that must have separete index.
Follow my filebeat.yml configuration:
#==================== Filebeat ==========================
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/*.log
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
#==================== Elasticsearch template setting ==========================
setup.template.enabled: true
setup.template.name: "normal"
setup.template.pattern: "normal-*"
setup.template.settings:
index.number_of_shards: 1
#============================== Dashboards =====================================
setup.dashboards.enabled: true
setup.dashboards.index: "normal-*"
#============================== Kibana =====================================
setup.kibana:
host: "192.168.0.191:5601"
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
hosts: ["192.168.0.191:9200"]
index: "%{[fields.log_type]}-%{[fields.machine]}-%{+yyyy.MM.dd}"
#================================ Processors =====================================
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
setup.ilm.enabled: false
`
NOTE:
1 - The Index Pattern on Kibana is created perfectly
2 - The Index Pattern on Elasticsearch is created perfectly after input some data
3 - The Syslog Module reads and shows vizualization on Syslog Dashboards perfectly by using the correct index pattern normal-*
4 - Netflow Dashboard Modules and other Pre Loaded Filebeat Dashboards have the same issue regards the reference to the filebeat-* pattern.
I would be very relieved if you could help me to find a solution for Index Pattern reference on these dashboards (Mainly the NetFlow Module) because they're so easy to manipulate, helpful, understandable for my current test environments. I am not afraid to create all of them in a template with the new normal-* pattern. I know how to do it but I am kind of lazy to recreate the whell this time since I am running different projects on the other hand, another reason is because the default ones have all I need so far.
Thanks for your project so far and I will be waiting a reply on this matter.
Regards,
