Filebeat setup: 400 Bad Request no handler found for uri [//_ilm/policy/filebeat]

[Nick95 (Nicolas)]

Hi,

I try to run a filebeat on a server. Elastic and Kibana are installed on another server and are pushed on 443 port via a nginx reverse proxy. Kibana GUI is working on the IP I set up (https with self-generated certificate)

When I run # sudo filebeat setup, I get a 400 bad request error:

*Exiting: failed to check for policy name 'filebeat': (status=400) {"error":"no handler found for uri [//_ilm/policy/filebeat] and method [GET]"}: 400 Bad Request: {"error":"no handler found for uri [//_ilm/policy/filebeat] and method [GET]*

I try to troubleshoot but I can’t see where is the issue.

Filebeat.yml On the filebeat server (only the uncommented lines)

# ============================== Filebeat inputs ===============================

filebeat.inputs:

- type: filestream

  id: my-filestream-id

  enabled: false

  paths:

    - /var/log/*.log

# ============================== Filebeat modules ==============================

filebeat.config.modules:

  path: ${path.config}/modules.d/*.yml

  reload.enabled: false

# ================================== Outputs ===================================

# ---------------------------- Elasticsearch Output ----------------------------

output.elasticsearch:

  hosts: ["<IP-ES-KIBANA>:443 /es"]

  ssl.certificate: "/ca/ca.crt"

  ssl.key: "/ca/ca.key"

  ssl.verification_mode: "none"

# ================================= Processors =================================

processors:

  - add_host_metadata:

      when.not.contains.tags: forwarded

  - add_cloud_metadata: ~

  - add_docker_metadata: ~

  - add_kubernetes_metadata: ~

Nginx conf on the Elastic/Kibana server

server {

  listen 443 ssl; 

  server_name <IP-ES-KIBANA>;

 

  ssl on;

  ssl_certificate /ca/ca.crt;

  ssl_certificate_key /ca/ca.key;

 

  location /kibana {

    proxy_pass http://localhost:5601;

      }

 

  location /es {

    proxy_pass http://localhost:9200/;

    #proxy_read_timeout 90;

  }

}

Version filebeat: 8.2.2 / Version Elastic search: 8.2.2

Can someone help please?

Thanks in advance,

Nick

[warkolm (Mark Walkom)]

Does it work if you talk directly to Elasticsearch/without nginx?

[Nick95 (Nicolas)]

Hi, yes if filebeat is installed locally behind the reverse proxy, it is working well and I can see logs on Kibana. It is when I try to set up filebeat on another server that I got the issue.

[warkolm (Mark Walkom)]

Then it is likely an nginx redirect issue.

[Nick95 (Nicolas)]

Hi,

To troubleshoot/simplify, I put everything on http and I still have the same issue.

I check in the log on my working filebeat locally and the correct message should be:

{"log.level":"info","@timestamp":"2022-08-10T09:31:43.699+0200","log.logger":"index-management.ilm","log.origin":{"file.name":"ilm/std.go","file.line":118},"message":"ILM policy filebeat exists already.","service.name":"filebeat","ecs.version":"1.6.0"}

So clearly filebeat on the remote server is failing to reach the correct kibana url. On remote server:

# curl -I http://<IP-ES-KIBANA/es give 200 OK

# curl -I http://<IP-ES-KIBANA/kibana give 302 FOUND

# curl -I http://<IP-ES-KIBANA/kibana/app/home give 200 OK

Using NGINX reverse proxy is pretty standard so I don’t understand where the issue come from.

Maybe something is missing in the configuration of filebeat.yml or nginx.conf ...