How to create aggregated field and further kibana dashboard

Hello, I created several filters in fluentd like below and have separate fields in Kibana as well: nginx_timestamp, DATA:nginx_log_level, nginx_message

Question: what is need to have aggregated field created nginx_log_level.keyword and further dashboard to be built?

I understand some mapping should be amendment, appreciate if you provide step-by-step guidance regarding this.

@type parser key_name log reserve_data true reserve_time true @type grok grok_failure_key grokfailure pattern %{TIMESTAMP_ISO8601:nginx_timestamp}" %{DATA:nginx_log_level} %{GREEDYDATA:nginx_message}

Hi @O_K,

If you are looking to visualize aggregated data for nginx_log_level.keyword, there are a couple of things you'll want to do:

  • Make sure you are sending the data from Fluentd to Elasticsearch with the correct nginx_log_level field name. Fluentd has an Elasticsearch output plugin you could use for this if you haven't found a solution already.
  • Update your ES mappings using the Mapping API. You'll probably want to either map nginx_log_level as keyword as shown here, or if you also need to perform full-text search, use multi-fields to map as both text and keyword as shown here.
  • Refresh your Index Pattern in Kibana (or create a new one if you haven't yet), and confirm the field is mapped as you expect.
  • Then you can create a new visualization with a terms aggregation on the nginx_log_level. This visualization can be saved and added to any dashboard. You could also apply filters to the dashboard itself to view data across all visualizations that's related to a particular log level.

The Kibana docs have a Build Your Own Dashboard tutorial which you may find helpful. It shows you how all of this looks end-to-end.

Hopefully this helps answer your question, but feel free to reach out if you need help with anything more specific!

Cheers

Luke

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.