Metricbeat daemonset modules fails

Elastic Stack Beats
1

I have configured following modules for metricbeat daemonset in AWS EKS

modules 
system.yml: |-
    - module: system
      period: 10s
      metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
        - core
        - diskio
        - socket
      processes: ['.*']
      process.include_top_n:
        by_cpu: 5      # include top 5 processes by CPU
        by_memory: 5   # include top 5 processes by memory

    - module: system
      period: 1m
      metricsets:
        - filesystem
        - fsstat
      processors:
      - drop_event.when.regexp:
          system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'

  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - node
        - system
        - pod
        - volume
        - event
        - container
      period: 10s
      enabled: true
      hosts: ["localhost:10250"]
      add_metadata: true
    - module: kubernetes
      enabled: true
      metricsets:
        - proxy
      hosts: ["localhost:10249"]
      period: 10s
    - module: kubernetes
      enabled: true
      metricsets:
        - controllermanager
      hosts: ["localhost:10252"]
      period: 10s
    - module: kubernetes
      enabled: true
      metricsets:
        - scheduler
      hosts: ["localhost:10251"]
      period: 10s

  traefik.yml: |-
    - module: traefik
      metricsets: ["health"]
      period: 10s
      hosts: ["localhost:8080"]

  linux.yml: |-
    - module: linux
      period: 10s
      metricsets:
        - "pageinfo"
        - "memory"
        # - ksm

        # - conntrack
        # - iostat
      enabled: true
      #hostfs: /hostfs

but seeing this in logs

2021-07-02T19:21:37.294Z	INFO	module/wrapper.go:259	Error fetching data for metricset traefik.health: failed to sample health: HTTP error 404 in : 404 Not Found
2021-07-02T19:21:37.309Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.scheduler: error getting processed metrics: error making http request: Get "http://localhost:10251/metrics": dial tcp 127.0.0.1:10251: connect: connection refused
2021-07-02T19:21:37.309Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.volume: error doing HTTP request to fetch 'volume' Metricset data: HTTP error 400 in : 400 Bad Request
2021-07-02T19:21:37.310Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: HTTP error 400 in : 400 Bad Request
2021-07-02T19:21:37.494Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.controllermanager: error getting processed metrics: error making http request: Get "http://localhost:10252/metrics": dial tcp 127.0.0.1:10252: connect: connection refused
2021-07-02T19:21:37.595Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.node: error doing HTTP request to fetch 'node' Metricset data: HTTP error 400 in : 400 Bad Request
2021-07-02T19:21:37.596Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.pod: error doing HTTP request to fetch 'pod' Metricset data: HTTP error 400 in : 400 Bad Request
2021-07-02T19:21:37.788Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.container: error doing HTTP request to fetch 'container' Metricset data: HTTP error 400 in : 400 Bad Request
2021-07-02T19:21:47.294Z	INFO	module/wrapper.go:259	Error fetching data for metricset traefik.health: failed to sample health: HTTP error 404 in : 404 Not Found
2021-07-02T19:21:47.309Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.scheduler: error getting processed metrics: error making http request: Get "http://localhost:10251/metrics": dial tcp 127.0.0.1:10251: connect: connection refused
2021-07-02T19:21:47.310Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: HTTP error 400 in : 400 Bad Request
2021-07-02T19:21:47.310Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.volume: error doing HTTP request to fetch 'volume' Metricset data: HTTP error 400 in : 400 Bad Request
2021-07-02T19:21:47.494Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.controllermanager: error getting processed metrics: error making http request: Get "http://localhost:10252/metrics": dial tcp 127.0.0.1:10252: connect: connection refused
2021-07-02T19:21:47.595Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.node: error doing HTTP request to fetch 'node' Metricset data: HTTP error 400 in : 400 Bad Request
2021-07-02T19:21:47.595Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.pod: error doing HTTP request to fetch 'pod' Metricset data: HTTP error 400 in : 400 Bad Request
2021-07-02T19:21:47.788Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.container: error doing HTTP request to fetch 'container' Metricset data: HTTP error 400 in : 400 Bad Request
2021-07-02T19:21:57.294Z	INFO	module/wrapper.go:259	Error fetching data for metricset traefik.health: failed to sample health: HTTP error 404 in : 404 Not Found
2021-07-02T19:21:57.309Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.scheduler: error getting processed metrics: error making http request: Get "http://localhost:10251/metrics": dial tcp 127.0.0.1:10251: connect: connection refused
2021-07-02T19:21:57.310Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: HTTP error 400 in : 400 Bad Request
2021-07-02T19:21:57.310Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.volume: error doing HTTP request to fetch 'volume' Metricset data: HTTP error 400 in : 400 Bad Request
2021-07-02T19:21:57.494Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.controllermanager: error getting processed metrics: error making http request: Get "http://localhost:10252/metrics": dial tcp 127.0.0.1:10252: connect: connection refused
2021-07-02T19:21:57.594Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.pod: error doing HTTP request to fetch 'pod' Metricset data: HTTP error 400 in : 400 Bad Request
2021-07-02T19:21:57.594Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.node: error doing HTTP request to fetch 'node' Metricset data: HTTP error 400 in : 400 Bad Request
2021-07-02T19:21:57.788Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.container: error doing HTTP request to fetch 'container' Metricset data: HTTP error 400 in : 400 Bad Request

metricbeat ClusterRole has following rules

ClusterRole Rules 
- apiGroups:
  - ""
  - extensions
  - apps
  resources:
  - namespaces
  - pods
  - services
  - events
  - deployments
  - nodes
  - nodes/stats
  - replicasets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - replicasets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - statefulsets
  - deployments
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/stats
  verbs:
  - get
- nonResourceURLs:
  - "/metrics"
  verbs:
  - get

Also tried to add https:// before localhost which resulted in following errors

2021-08-10T18:52:04.250Z	ERROR	[kubernetes.container]	container/container.go:93	error making http request: Get "https://localhost:10250/stats/summary": x509: certificate signed by unknown authority
2021-08-10T18:52:04.250Z	ERROR	[kubernetes.node]	node/node.go:95	error making http request: Get "https://localhost:10250/stats/summary": x509: certificate signed by unknown authority
2021-08-10T18:52:12.852Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.scheduler: error getting processed metrics: error making http request: Get "https://localhost:10251/metrics": dial tcp 127.0.0.1:10251: connect: connection refused
2021-08-10T18:52:13.162Z	INFO	module/wrapper.go:259	Error fetching data for metricset traefik.health: failed to sample health: HTTP error 404 in : 404 Not Found
2021-08-10T18:52:13.569Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.controllermanager: error getting processed metrics: error making http request: Get "https://localhost:10252/metrics": dial tcp 127.0.0.1:10252: connect: connection refused
2021-08-10T18:52:13.569Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.proxy: error getting processed metrics: error making http request: Get "https://localhost:10249/metrics": http: server gave HTTP response to HTTPS client
2021-08-10T18:52:14.250Z	ERROR	[kubernetes.pod]	pod/pod.go:94	error making http request: Get "https://localhost:10250/stats/summary": x509: certificate signed by unknown authority
2021-08-10T18:52:14.250Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.volume: error doing HTTP request to fetch 'volume' Metricset data: error making http request: Get "https://localhost:10250/stats/summary": x509: certificate signed by unknown authority
2021-08-10T18:52:14.250Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: error making http request: Get "https://localhost:10250/stats/summary": x509: certificate signed by unknown authority
2021-08-10T18:52:14.250Z	ERROR	[kubernetes.container]	container/container.go:93	error making http request: Get "https://localhost:10250/stats/summary": x509: certificate signed by unknown authority
2021-08-10T18:52:14.250Z	ERROR	[kubernetes.node]	node/node.go:95	error making http request: Get "https://localhost:10250/stats/summary": x509: certificate signed by unknown authority

is it possible to get it working on AWS EKS ?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.