Hello, I created several filters in fluentd like below and have separate fields in Kibana as well: nginx_timestamp, DATA:nginx_log_level, nginx_message
Question: what is need to have aggregated field created nginx_log_level.keyword and further dashboard to be built?
I understand some mapping should be amendment, appreciate if you provide step-by-step guidance regarding this.
If you are looking to visualize aggregated data for nginx_log_level.keyword, there are a couple of things you'll want to do:
Make sure you are sending the data from Fluentd to Elasticsearch with the correct nginx_log_level field name. Fluentd has an Elasticsearch output plugin you could use for this if you haven't found a solution already.
Update your ES mappings using the Mapping API. You'll probably want to either map nginx_log_level as keywordas shown here, or if you also need to perform full-text search, use multi-fields to map as both text and keywordas shown here.
Refresh your Index Pattern in Kibana (or create a new one if you haven't yet), and confirm the field is mapped as you expect.
Then you can create a new visualization with a termsaggregation on the nginx_log_level. This visualization can be saved and added to any dashboard. You could also apply filters to the dashboard itself to view data across all visualizations that's related to a particular log level.
The Kibana docs have a Build Your Own Dashboard tutorial which you may find helpful. It shows you how all of this looks end-to-end.
Hopefully this helps answer your question, but feel free to reach out if you need help with anything more specific!
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.