Metricbeat Timeout Errors while scraping state metricset from kube-state-mertics

Elastic Stack Beats
1

I am running metricbeat deployment and daemonset using elasticsearch helm chart. In my metricbeat deployment I am collecting state metrics from the kube-state-metrics API from an openshift cluster and experience following errors from time to time:

2021-08-19T08:31:50.189Z	ERROR	[kubernetes.state_deployment]	state_deployment/state_deployment.go:98	decoding of metric family failed: context deadline exceeded (Client.Timeout or context cancellation while reading body)

2021-08-19T08:33:39.893Z	INFO	module/wrapper.go:259	Error fetching data for metricset kubernetes.state_container: error getting event: decoding of metric family failed: context deadline exceeded (Client.Timeout or context cancellation while reading body)

At the same time I see that some state metrics are getting indexed in elasticsearch and some I am missing. Maybe it is related to the timeouts I see in my metricbeat deployment

Thanks

2

I have inceased the ressources for my metricbeat deployment and it now seems to have no timeout errors

3

Could you elaborate on what resources you increased?

4

I have increased the metricbeat deployment container resources to 1 CPU and 1Gi memory:

  resources:
    requests:
      cpu: "1"
      memory: "1Gi"
    limits:
      cpu: "1"
      memory: "1Gi"
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.