Filebeat cannot connect to kibana: error loading index pattern: parsing kibana response

Elastic Stack Beats
1

I have set up Kibana and Elasticsearch and they work fine so far.

I have also set up Filebeat on another aws ec2 instance to send logs to Elasticsearch and monitor from Kibana

But for some reason Filebeat cannot connect to Kibana. filebeat setup -e command logs an error for Kibana

stack info:
filebeat version 7.13.4 (amd64), libbeat 7.13.4
kibana v 7.13.4
elasticsearch v 7.13.4

filebeat setup -e log info:

2021-07-22T12:28:26.992+0900	INFO	template/load.go:228	Existing template will be overwritten, as overwrite is enabled.
2021-07-22T12:28:28.440+0900	INFO	template/load.go:131	Try loading template filebeat-7.13.4 to Elasticsearch
2021-07-22T12:28:29.206+0900	INFO	template/load.go:123	template with name 'filebeat-7.13.4' loaded.
2021-07-22T12:28:29.206+0900	INFO	[index-management]	idxmgmt/std.go:297	Loaded index template.
2021-07-22T12:28:29.217+0900	INFO	[index-management.ilm]	ilm/std.go:121	Index Alias filebeat-7.13.4 exists already.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
2021-07-22T12:28:29.217+0900	WARN	[cfgwarn]	tlscommon/config.go:105	DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0
2021-07-22T12:28:29.217+0900	INFO	kibana/client.go:119	Kibana url: https://kibana.myurl.com:443
2021-07-22T12:28:31.177+0900	INFO	kibana/client.go:119	Kibana url: https://kibana.myurl.com:443
2021-07-22T12:28:31.238+0900	ERROR	instance/beat.go:989	Exiting: 1 error: error loading index pattern: parsing kibana response </html>nter>nginx/1.18.0 (Ubuntu)</center>h1></center>d>esponse: <html>
Exiting: 1 error: error loading index pattern: parsing kibana response: invalid character '<' looking for beginning of value. Response </html>nter>nginx/1.18.0 (Ubuntu)</center>h1></center>d>

Could anyone please help me figure this out?

2

HI @outranker :slightly_smiling_face:

Try to curl -XGET http://your_kibana_instance:5601/api/status to get a response. But it seems from those logs you have an nginx that might be affecting your connection

3

Hi @Mario_Castro , thank you for your reply!

Get request to /api/status route returns a json result containing status info. I think Kibana is working fine because I can access it.

Also, since Kibana is being served behind nginx reverse proxy i did not append 5601 port number to the curl get request you mentioned.

I am guessing this is related nginx/network but cannot figure out exactly what it is.
The url for Kibana is kibana.javohirmirzo.com

4

How does your Filebeat config looks like? One thing you said that I realized now and I didn't the first time. You are trying to connect to Kibana from Filebeat to fetch Kibana logs. To do so you must configure the Filebeat module of Kibana.

Just in case, and forgive me for being a bit basic, you don't send your logs to Kibana but to elasticsearch and you use kibana to visualize them. If you want to fetch Kibana logs to later visualize them on Kibana too, Filebeat must be deployed on the same instance of Kibana and read those log files (you can't access them by http)

5

This is Filebeat config:

# ============================== Filebeat inputs ===============================
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/ubuntu/my-project/logs/*


# filestream is an experimental input. It is going to replace log input in the future.
- type: filestream
  enabled: false
  paths:
    - /var/log/*.log


# ============================== Filebeat modules ==============================
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true


# ======================= Elasticsearch template setting =======================
setup.template.settings:
  index.number_of_shards: 1


# =================================== Kibana ===================================
setup.kibana:
        host: "https://kibana.javohirmirzo.com:443"
        username: "username"
        password: "my-password"


# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
        hosts: ["http://elasticsearch.javohirmirzo.com:80"]
        username: "username"
        password: "my-password"


# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

If I open port 5601 for Kibana on the host and configure Filebeat to that url i.e ip address plus 5601 port everything works fine.

In the beginning the problem was caused by Filebeat's built-in Kibana dashboard being enabled but when i enable and disable Kibana module in Filebeat back-and-forth and test it the result is the same.

You are trying to connect to Kibana from Filebeat to fetch Kibana logs.

I am not sure if this is true but as far as i know if Kibana settings is not present in Filebeat config
file, then Filebeat tries to access Kibana on localhost so Kibana settings should explicitly be given in the config file. Also I am not planning to make use of Kibana instance logs for now and this connection to Kibana from Filebeat is unintentional. I guess this is the predetermined behaviour of Filebeat?

you don't send your logs to Kibana but to elasticsearch and you use kibana to visualize them.

Yes, I want to send my logs to Elasticsearch and visualize them in Kibana, if my settings reflect otherwise please help me figure this out :slightly_smiling_face:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.