Metricbeat stuck on "net/http: request canceled (Client.Timeout exceeded while awaiting headers)"

Elastic Stack Beats
1

Hi!
I'm having trouble with loading template into Elasticsearch.

My cluster currently contains only one VM. Since i finished it's set up i moved into shipping data from the external VM via Metricbeat... and it didn't went as planned.
I can't load a template or event ship some data. The weirdest part of it is that
sometimes it works (5% of attemps), but mainly not.
From Metricbeat the only error im getting is:

May 17 14:23:09 HOSTNAME-X metricbeat[24832]: 2021-05-17T14:23:09.740+0200 ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://MYELASTIC:9200)): Connection marked as failed because the onConnect callback failed: error loading template: could not load template. Elasticsearch returned: couldn't load template: Put "https://MYELASTIC:9200/_index_template/metricbeat-HOSTNAME-X-7.12.1": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Even if it works by accident the next thing i get is:

May 17 13:29:51 HOSTNAME-X metricbeat[29339]: 2021-05-17T13:29:51.334+0200 ERROR [elasticsearch] elasticsearch/client.go:224 failed to perform any bulk index operations: Post "https://MYELASTIC:9200/_bulk": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

The:

metricbeat setup --index-management

end with the same result as loading template via daemon at its start.

Are those connection related issuses? Or i need to look deeper into logs? Does Metricbeat requires two ways connection? Elasticsearch < -- > Metricbeat ?

I have checked logs on Elasticsearch service but there was no logs related to issue.

2

I have tested the connection from HOSTNAME-X into Elasticsearch, and its working:

curl https://MYELASTIC:9200 --cacert /etc/metricbeat/elasticsearch-ca.pem 
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}

So what can be cause of request canceled (Client.Timeout exceeded while awaiting headers) ?

3

Can the cause be a too slow output connection?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.