Could not communicate with Checking API

[dmitryyankowski]

Hi guy's i'm setting up an elastic-agent on a Windows machine... and looking through the logs i'm seeing the following error:

{"log.level":"error","@timestamp":"2021-03-03T09:41:49.593-0500","log.origin":{"file.name":"application/fleet_gateway.go","file.line":187},"message":"Could not communicate with Checking API will retry, error: could not decode the response

"message":"Could not communicate with Checking API will retry, error: could not decode the response

I can make a request just fine via the browser or through CURL just fine at the https://kibana.mydomain.com/api/fleet/agents/e416bec0-7c17-11eb-a501-6d52f7426339/checkin URL... so not sure what the issue is.

^ I can make a test POST request to the endpoint just fine... not sure why elastic-agent is throwing the error.

Also i'm receiving Elasticsearch data just fine to my Elasticsearch endpoint to https://elasticsearch.mydomain.com ... it's just that Kibana can't make this request and then the ElasticAgent shuts down and stops working

Note: I am using Cloudflare and nginx as a reverse proxy to both my Kibana and ElasticSearch endpoints.

Anyone have any ideas what the issue may be?

[nchaulet (Nicolas Chaulet)]

Hi @dmitryyankowski

Did you configure the Kibana URL and Elasticsearch URL in the settings panel of Fleet, it could be the issue?

[dmitryyankowski]

@nchaulet thanks for getting back to me so quick

Kinda still learning the ELK Stack so bear with me here lol.
This is what my panel look's like:

I can see that a POST request is being made to my elasticsearch endpoint... but when I do a CURL (as shown in my original post) or go to the URL in the browser it works just fine?

Note: I'm reverse proxying the Kibana and ElasticSearch ports to 443 for each service. I had to put the port on the end of both the Kibana URL and ElasticSearch URL, or else it would try to use the default 5601 and 9200 ports.

Is there something else i'm missing?

Also please look at this... ElasticAgent (on the right) can't send the request for some reason.. while I make make the same CURL request on the left?

@nchaulet i'm able to install the elastic agent and i'm receiving logs with Elasticsearch.. what's stopping that CheckIn request? Do I need to provide credentials somewhere for it?

After awhile... it ends up killing the agent

[nchaulet (Nicolas Chaulet)]

Fleet use long polling for that checkin call, it's possible that you reverse proxy has some timeout issue here, by default the long polling call is configured to be 5 minutes.

[dmitryyankowski]

Ah! I have a pretty default Nginx configuration I think... so i'll set the timeout to 10 minutes then Thank you!

[dmitryyankowski]

@nchaulet Any ideas what the long polling config would look like in NGINX?

Here's my current config. I'm still getting timeouts though

[dmitryyankowski]

Ah!

CloudFlare will also wait 100 seconds for a HTTP response from your server before you will see a 524 timeout error.

How do I update the checking timeout from 5 minutes to say 30 seconds?

[dmitryyankowski]

Figured it out

In my kibana.yml file I added the following:

xpack.fleet.agents.pollingRequestTimeout: 30000

No more timeouts!