Filebeat not able to connect to kibana hosted in another server

Elastic Stack Beats
1

Hi, I have hosted elasticsearch and kibana in one server and filebeat and my application in another server. I want to send my application logs to elasticsearch via filebeat. I have configured my filebeat.yml to IP of elasticsearch and kibana. When i do curl GET on elasticsearch, im getting proper response, but to kibana i'm getting error as "curl: (35) error:0A00010B:SSL routines::wrong version number"

Also when i do "./filebeat setup -e" i'm getting below error.

"message":"Exiting: error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory <path to elk_stack>/elk_stack/filebeat-8.14.3-linux-x86_64/kibana: failed to import Kibana index pattern: 1 error: error loading index pattern: returned 413 to import file: error extracting JSON for error response: invalid character '<' looking for beginning of value. Response: <html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body>\r\n<center>413 Request Entity Too Large</center>\r\n<hr><center>nginx/1.24.0 (Ubuntu)</center>\r\n</body>\r\n</html>\r\n","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory <path to elk_stack>/elk_stack/filebeat-8.14.3-linux-x86_64/kibana: failed to import Kibana index pattern: 1 error: error loading index pattern: returned 413 to import file: error extracting JSON for error response: invalid character '<' looking for beginning of value. Response: <html>
<head><title>413 Request Entity Too Large</title></head>
<body>
<center>413 Request Entity Too Large</center>
<hr><center>nginx/1.24.0 (Ubuntu)</center>
</body>
</html>

Versions :
Elasticsearch : 8.14.3
Kibana : 8.14.3
Filebeat : 8.14.3

Can anyone please help on this issue ?

Thanks,
Nanditha

2

Hi @Nanditha

Share your filebeat.yml please.

How did you install elasticsearch and Kibana

3

Hi @stephenb, thank you for your replay.

I have installed elasticsearch and kibana via debian package, referring to below official elasticsearch links.

one more thing I missed mentioning in my previous post, my elastic and kibana is in AWS EC2 instance and my filebeat is inside a docker container. I have allowed all/any connection to my container using this command,
docker run -it --net=host imageid bash

Here is my filebeat.yml :

# ============================== Filebeat inputs ===============================

filebeat.inputs:
- type: log
  id: my-filestream-id
  enabled: true
  paths:
    - /nanditha/evaluation/logs/*
    - /isa/evaluation/logs/exec_time.log

  multiline.pattern: '^\[\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d+\] \[INFO\] \[Logged into as user\]'
  multiline.negate: true
  multiline.match: after
  multiline.timeout: 30s
  multiline.flush_pattern: '\[process shutting down\]'

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1

# =================================== Kibana ===================================
setup.kibana:
  host: "x.x.x.x:80"
  username: "x"
  password: "x"

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  hosts: ["x.x.x.x:9200"]
  protocol: "https"
  username: "x"
  password: "x"
  ssl.verification_mode: none

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# ================================== Logging ===================================
logging.level: info
logging.to_files: true
logging.files:
  path: /isa/containers/eval_pipeline_mlflow/elk_stack/filebeat-8.14.3-linux-x86_64/logs
  name: filebeat
  keepfiles: 7
  permissions: 0640
4

Looks like you're running an nginx proxy in front of kibana. I suspect that may be your issue here. You'll want that setup to go directly to Kibana.

5

Hi @stephenb,
Can you please help me with the steps/any website i can refer to, if you know.

6

Steps for what?

In filebeat.yml you need to set

Directly to the Kibana host port....

7

Thank you. The issue is resolved. I disabled nginx and set my kibana port to default(5601) and updated same in my filebeat.yml. Everything works now.

8

If you would like to keep using nginx you just need to identify what path you've put the kibana api behind, like http://nginx:80/kibanaapi and then set setup.kibana.path to that path for example:
setup.kibana.path: kibanaapi

9

Sure, Thank you.