Metricbeat index missing field format

[ELK_GB]

Hi,

I have metricbeat 5.4 deployed on multiple sources and data is being sent to ES as follows:

Source (beat) > Kafka > LS 5.4 > ES 5.4 (the index name is metricbeat-* the default)

My issue is for example any ".pct" or "bytes" fields are not being formatted as percentage nor byte; and I need to do that manually, in Kibana.
Even If i do it manually once a new metricbeat index is created my formatting is gone again!

I looked at few suggested solutions, one was suggesting that once the metrics are not being sent directly from the beat to ES it will lose the index template/formatting; and the template needs to be pushed manually.

So I pushed it manually; and now I have a punch of metric fields for different modules that I dont use such as kafka, mysql, zookeeper...etc. and the formatting issue is not resolved it didn't change.

In summary i need to achieve two things:

1. Get my field formatting done automatically for data coming from metricbeat and indexed as metricbeat-* (the default). Am I missing anything at the logstash ES output ?

2. Clean up all the unused fields that are now available in Kibana/index as a result of me pushing a metricbeat template manually.

[ruflin]

There are two things here: index template for elasticsearch and index pattern for Kibana. The one to make .pct look like percentages in Kibana is the index pattern. If you don't connect to ES / KB directly, you need to load these manually. For 5.6 you can see here how you load the dashboards / index patterns: https://www.elastic.co/guide/en/beats/libbeat/5.6/import-dashboards.html

For the second part, there is no work around to this. The reason is that we don't know in advance which fields you will use so we push all of them. You "could" manually modify the template and index pattern but you would need to do that for each version. Any specific problem with having the fields in there?

[ELK_GB]

@ruflin

I appreciate your response !

What I ended up doing over the weekend is manually changing the format for all pct and bytes fields, for the metricbeat index we are using. I also have deleted the metricbeat template I pushed manually earlier and started fresh, so that took care of the extra fields I don't use. Now just letting LS push the metrics to ES and let the process decide on the ES template.

For the Kibana index pattern I will test that and see what I get, may not need it at this point but very useful to try.

One more question for you, so if send metrics directly from the beat to ES it will push all the needed ES and Kibana templates automatically ?

[ruflin]

It will push the ES template automatically (for 5.x and 6.x). For 6.x you can run ./metricbeat setup to push the Kibana index pattern and dashboards. In case it's not running on localhost, you have to configure setup.kibana.* accordingly. For 5.x you need to use the import_dashboards scripts.