HTTP error 403 in pod - Metricbeat in Openshift

I'm trying to run the Kubernetes module in the Openshift metricbeat configuration. When I consult in kibana, it shows me the following error:

      "key": "HTTP error 403 in volume: 403 Forbidden",
      "doc_count": 348,
      "NAME": {
        "doc_count_error_upper_bound": 0,
        "sum_other_doc_count": 0,
        "buckets": [
            "key": "",
            "doc_count": 35

I have followed the steps indicated in the documentation

this is the configuration I have right now assigned to the kubernatorial module:

  • module: kubernetes
    - node
    - system
    - pod
    - container
    - volume
    period: 10s
    host: {NODE_NAME} hosts: ["https://{NODE_NAME}:10250"]
    bearer_token_file: /var/run/secrets/token
    - "/var/run/secrets/"

The assigned token is added in the Daemonset taken from the secret of the service account metricbeat. Initially, I assigned the service account cluster-admin permissions.

As a test, perform the following curl inside the daemonset pod with the token and the CA configured in the Metricbeat kubernatorial module and give a 200 OK showing the desired metrics of the kubernatorial module:

curl -H "Authorization: Bearer (token service account metricbeat)" --cacert /var/run/secrets/ -v https://${NODE_NAME}:10250/stats/summary -k

< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Fri, 17 Jul 2020 10:37:53 GMT
< Transfer-Encoding: chunked
{ [data not shown]
"node": {
"nodeName": "",
"systemContainers": [
"name": "kubelet",
"startTime": "2020-07-07T06:12:38Z",
"cpu": {
"time": "2020-07-17T10:37:48Z",
"usageNanoCores": 85288876,
"usageCoreNanoSeconds": 56802787834732

Is there anything I'm leaving along the way?


Something is wrong with your certificate.
When you are doing the curl you add -k flag which enables the insecure connection.

You can configure the module to add this insecure flag too with adding ssl.verification_mode: "none" , see


It's weird that curl works with the certificate :thinking:.

Could you try with:

      hosts: ["https://${NODE_NAME}:10250"]
      bearer_token_file: /var/run/secrets/
      ssl.verification_mode: "none"


Also could you check what is logged in Metricbeat logs?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.