HTTP error 403 in pod - Metricbeat in Openshift

Elastic Stack Beats
1

Hi.
I'm trying to run the Kubernetes module in the Openshift metricbeat configuration. When I consult in kibana, it shows me the following error:

      "key": "HTTP error 403 in volume: 403 Forbidden",
      "doc_count": 348,
      "NAME": {
        "doc_count_error_upper_bound": 0,
        "sum_other_doc_count": 0,
        "buckets": [
          {
            "key": "node.name.es:10250",
            "doc_count": 35
          },

I have followed the steps indicated in the documentation Run Metricbeat on Kubernetes | Metricbeat Reference [8.11] | Elastic

this is the configuration I have right now assigned to the kubernatorial module:

  • module: kubernetes
    metricsets:
    - node
    - system
    - pod
    - container
    - volume
    period: 10s
    host: ${NODE_NAME}
    hosts: ["https://${NODE_NAME}:10250"]
    bearer_token_file: /var/run/secrets/token
    ssl.certificate_authorities:
    - "/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt"

The assigned token is added in the Daemonset taken from the secret of the service account metricbeat. Initially, I assigned the service account cluster-admin permissions.

As a test, perform the following curl inside the daemonset pod with the token and the CA configured in the Metricbeat kubernatorial module and give a 200 OK showing the desired metrics of the kubernatorial module:

curl -H "Authorization: Bearer (token service account metricbeat)" --cacert /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt -v https://${NODE_NAME}:10250/stats/summary -k

< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Fri, 17 Jul 2020 10:37:53 GMT
< Transfer-Encoding: chunked
<
{ [data not shown]
{
"node": {
"nodeName": "node.name.es:10250",
"systemContainers": [
{
"name": "kubelet",
"startTime": "2020-07-07T06:12:38Z",
"cpu": {
"time": "2020-07-17T10:37:48Z",
"usageNanoCores": 85288876,
"usageCoreNanoSeconds": 56802787834732
},

Is there anything I'm leaving along the way?

2

Hi!

Something is wrong with your certificate.
When you are doing the curl you add -k flag which enables the insecure connection.

You can configure the module to add this insecure flag too with adding ssl.verification_mode: "none" , see https://github.com/elastic/beats/blob/4ce680c62f2c9b66bba4594f4a8569965a5f22a1/deploy/kubernetes/metricbeat/metricbeat-daemonset-configmap.yaml#L81.

C.

4

It's weird that curl works with the certificate :thinking:.

Could you try with:

      hosts: ["https://${NODE_NAME}:10250"]
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      ssl.verification_mode: "none"

?

Also could you check what is logged in Metricbeat logs?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.