All Kibana is doing is taking the values from the prometheus exporter and doing the math to display... In my experience, It is often pretty hard to get exact comparisons from a CMD Line tool and and a metrics collections and visualizations. The Counter Rate in Kibana is pretty tested I use it all the time for network metrics and it is pretty solid (not to say there couldn't be an issue)
You might need to do some deep reading on How kubectl is displaying data vs how the data is collected and reported by prometheus. I suspect there is nuances... example just some nuance on the memory collection (not cpu) but similar.
Me? I would probably lean towards the prometheus collected metrics as that has such a wide user base, (assuming everything is configured correctly) and many folks monitor their K8s with prometheus collectors if there were issues I suspect they would be reported and fixed.
I don't have a K8s cluster up and running right now so I can not compare.
Ok let's back up a bit... I am / was not reading carefully .... apologies
I was focused on showing you the correct way to calculate a rate... What we just calculated was the Rate of container_cpu_usage_total i.e the rate of CPU consumption (AND we should have SUM up the rates with a Series agg for all the Containers anyways which we did not do yet), not the percent CPU which I now understand what you wan
So you want
A) The Total CPU Percent for All Containers Per Node?
B) The CPU Percent Per Container per Node?
So I think we need to do some re-thinking and yes you can do it, but it is going to take more work. (Of course this would most of this would be done for you automatically if you used metricbeat collection )
I am not a prometheus expert looks like something like these would be the types calculations we would need to do..
Pick A) or B) above and then find the other fields like total_cores and the collection rate etc.
Then perhaps I can help. If it is for all container we will need to sum the container_cpu_usage_total if for each container then will need to be broken down by container.
Then it will looks something like this (This is not correct just and example) We will need to do a bucket script etc. and use the math from the Stack Overflow etc.