I'm using the Zeek module in filebeat to ingest logs. By default, the module imports around 5000 fields into kibana and I'm wondering if it's best practice to cut that list down or not.
I'm noticing my dashboard is running quite slow and I'm wondering if the sheer amount of fields would slow things down substantially or not. In actuality I'm only using about 100 of the fields.
I have spent a few weeks investigating this and yes, too many fields is a problem for Kibana and is a performance issue in all versions. There is some good news here:
If you use Fleet/Elastic agent to install modules, they use small datastreams for each module instead of a huge mapping. This is a major performance improvement.
I coordinated with a few teams on the Kibana side to improve performance of large mappings in 7.13- we were seeing particularly bad performance in 7.9 to 7.12. More improvements are coming in future versions as we are starting to monitor this more actively.
If you can't upgrade or change your method of data ingestion, the only way to improve this issue is to:
Thanks for the response @wylie .
So does 7.13 show a big improvement in performance for this or has any change in 7.13 been made for this and if so, what?
In terms of using a fleet/elastic agent, how would I migrate to using this from what I'm currently using with filebeat? I would still need to reindex, correct?
@seanziee There were a number of small performance changes merged for 7.13, but based on what I could measure it adds up to a significant improvement in dashboards/visualizations. Not all parts of Kibana are going to get the same improvement though.
The deployment model for fleet/elastic agent is a little different from filebeat, I'd encourage you to read the docs about how it works. I am personally not sure if it would copy any files that were previously copied by another filebeat instance- all I can say is that Elastic agent uses fewer mappings.
So I've looked into fleet before and seemed a bit confusing and not really sure of the benefit I get, especially because my filebeat config has lots of different sources and processors to clean up the data so I'd need to figure all of that out for fleet.
So then it seems that potentially I would get the same benefit by going into the filebeat template and adjusting that to what I need. Is that correct or is there any other benefit?
I'm also unable to locate a list of the available integrations that fleet offers,do you know where one is? I need to see if zeek is on that list, and if so potentially I can get a list of the mappings that the integration offers.
For the workaround, you can't just change the template, you also need to reindex all the data into the new template. It is not possible to delete mappings without reindexing.
We have a document listing all the benefits of fleet and agent here: Fleet and Elastic Agent overview | Fleet User Guide [7.13] | Elastic
The list of integrations is only available from the Kibana UI. It looks like Zeek is supported, but you'd need to test it out.
Great, thanks. Yes, I understand I'd need to reindex, but I think that would be the case with both solutions.
Is there a github that lists what the datastream uses to create the zeek datastream?
This is not my area of expertise, but I am pretty sure that the agent is using the same shared code to collect files as filebeat, which is why it has the same module support. The main difference is that the agent automatically configures the datastreams based on what you're using.
Ok thanks for your help 
@wylie , I adjusted the mappings so now I have 700 fields instead of 5k and sadly things are still going very slow for version 7.11. In the past I've upgraded and haven't seen much of a difference, and I'm not terribly convinced that this upgrade would help much. It's just a lot of risk for breaking each time we upgrade.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.