Filebeat cannot connect to kibana: error loading index pattern: parsing kibana response

I have set up Kibana and Elasticsearch and they work fine so far.

I have also set up Filebeat on another aws ec2 instance to send logs to Elasticsearch and monitor from Kibana

But for some reason Filebeat cannot connect to Kibana. filebeat setup -e command logs an error for Kibana

stack info:
filebeat version 7.13.4 (amd64), libbeat 7.13.4
kibana v 7.13.4
elasticsearch v 7.13.4

filebeat setup -e log info:

2021-07-22T12:28:26.992+0900	INFO	template/load.go:228	Existing template will be overwritten, as overwrite is enabled.
2021-07-22T12:28:28.440+0900	INFO	template/load.go:131	Try loading template filebeat-7.13.4 to Elasticsearch
2021-07-22T12:28:29.206+0900	INFO	template/load.go:123	template with name 'filebeat-7.13.4' loaded.
2021-07-22T12:28:29.206+0900	INFO	[index-management]	idxmgmt/std.go:297	Loaded index template.
2021-07-22T12:28:29.217+0900	INFO	[index-management.ilm]	ilm/std.go:121	Index Alias filebeat-7.13.4 exists already.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
2021-07-22T12:28:29.217+0900	WARN	[cfgwarn]	tlscommon/config.go:105	DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0
2021-07-22T12:28:29.217+0900	INFO	kibana/client.go:119	Kibana url: https://kibana.myurl.com:443
2021-07-22T12:28:31.177+0900	INFO	kibana/client.go:119	Kibana url: https://kibana.myurl.com:443
2021-07-22T12:28:31.238+0900	ERROR	instance/beat.go:989	Exiting: 1 error: error loading index pattern: parsing kibana response </html>nter>nginx/1.18.0 (Ubuntu)</center>h1></center>d>esponse: <html>
Exiting: 1 error: error loading index pattern: parsing kibana response: invalid character '<' looking for beginning of value. Response </html>nter>nginx/1.18.0 (Ubuntu)</center>h1></center>d>

Could anyone please help me figure this out?

HI @outranker :slightly_smiling_face:

Try to curl -XGET http://your_kibana_instance:5601/api/status to get a response. But it seems from those logs you have an nginx that might be affecting your connection

Hi @Mario_Castro , thank you for your reply!

Get request to /api/status route returns a json result containing status info. I think Kibana is working fine because I can access it.

Also, since Kibana is being served behind nginx reverse proxy i did not append 5601 port number to the curl get request you mentioned.

I am guessing this is related nginx/network but cannot figure out exactly what it is.
The url for Kibana is kibana.javohirmirzo.com

How does your Filebeat config looks like? One thing you said that I realized now and I didn't the first time. You are trying to connect to Kibana from Filebeat to fetch Kibana logs. To do so you must configure the Filebeat module of Kibana.

Just in case, and forgive me for being a bit basic, you don't send your logs to Kibana but to elasticsearch and you use kibana to visualize them. If you want to fetch Kibana logs to later visualize them on Kibana too, Filebeat must be deployed on the same instance of Kibana and read those log files (you can't access them by http)