Filebeat 9.1.5 → Elasticsearch 9.1.5: /_license?human=false returns 404 {}, Filebeat fails with “onConnect callback failed: 404 {}”
Environment
-
Elasticsearch: 9.1.5 (Deb package,
build_flavor: default) -
Kibana: 9.1.5
-
Filebeat: 9.1.5
-
Cluster: 3 nodes, TLS enabled (auto-generated
http_ca.crt), security enabled -
OS: Linux (Debian/Ubuntu family)
-
Proxy / LB in front of ES: No (direct connection)
Symptom
Filebeat connects to ES but immediately fails during the compatibility check:
Failed to connect to backoff(elasticsearch(https://10.0.100.61:9200)):
Connection marked as failed because the onConnect callback failed:
could not connect to a compatible version of Elasticsearch: 404 Not Found: { }
With debug:
... esclientleg.(*Connection).Request GET https://10.0.100.61:9200/_license?human=false
→ 404 {}
What I verified
1) ES root endpoint OK
curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://10.0.100.61:9200/
# 200 OK, shows version 9.1.5 (default)
2) The exact endpoint Filebeat calls returns 404
curl -i --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic \
'https://10.0.100.61:9200/_license?human=false'
HTTP/1.1 404 Not Found
X-elastic-product: Elasticsearch
content-type: application/json
content-length: 4
{ }
(Same result on https://127.0.0.1:9200/_license?human=false locally on the node.)
3) Cluster is healthy
curl -k -u elastic https://10.0.100.61:9200/_cluster/health?pretty
# status: green
4) X-Pack modules are present (this is the default distribution)
ls -1 /usr/share/elasticsearch/modules | grep -i x-pack | wc -l
# many x-pack-* modules present
curl -s --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic \
"https://10.0.100.61:9200/_cat/plugins?v"
# empty (understandable: x-pack is a module, not a plugin)
5) No custom filtering for HTTP paths
-
There is no
http.filter.pathor similar inelasticsearch.yml. -
Nothing sits in front of ES (no nginx/haproxy).
6) Node/cluster license settings
curl -s --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic \
"https://127.0.0.1:9200/_nodes/settings?filter_path=nodes.*.settings.xpack.license"
# {}
curl -s --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic \
"https://127.0.0.1:9200/_cluster/settings?include_defaults=true&filter_path=defaults.xpack.license,persistent.xpack.license,transient.xpack.license" | jq .
# shows defaults with self_generated.type=basic
7) Filebeat test output (TLS + fingerprint)
filebeat test output \
-E output.elasticsearch.hosts=["https://10.0.100.61:9200"] \
-E output.elasticsearch.username=elastic \
-E output.elasticsearch.password='*****' \
-E output.elasticsearch.ssl.ca_trusted_fingerprint='598D8B23...F481'
# TLS handshake OK, Ping 200, then GET /_license?human=false → 404 {}
Config snippets (sanitized)
/etc/filebeat/filebeat.yml
setup.kibana:
host: "http://10.0.100.61:5601"
output.elasticsearch:
hosts: ["https://10.0.100.61:9200"]
preset: balanced
ssl.ca_trusted_fingerprint: "*****"
username: "elastic"
password: "*****"
filebeat.inputs:
- type: syslog
enabled: true
protocol.udp:
host: "0.0.0.0:5514"
tags: ["mikrotik","syslog"]
/etc/elasticsearch/elasticsearch.yml
cluster.name: Firewall
node.name: node-1
node.roles: [master, data_hot, ingest, data_content]
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
transport.port: 9300
discovery.seed_hosts: ["10.0.100.61:9300","10.0.100.62:9300","10.0.100.63:9300"]
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["node-1","node-2","node-3"]
http.host: 0.0.0.0
Timeline / extra context
-
This setup was working earlier and then suddenly stopped working for Filebeat outputs.
-
Fleet-Server/Agent are healthy and ES/Kibana work normally. Only Filebeat’s ES output fails because
/_license?human=falsereturns 404{}.
Questions for the Elastic team
-
In Elasticsearch 9.1.5 (default flavor), should
GET /_license?human=falsealways be available? Under what conditions would it return 404 (instead of 200/401/403)? -
Is there any setting (cluster or node) that can disable the license route without logging an error/warning? I don’t have
http.filter.pathand there is no reverse proxy. -
Could this be a regression/bug in 9.1.5 where the license endpoint is not registered in some startup order? If so, which logs/diagnostics can I capture to help?
-
Any suggested workaround to let Filebeat skip the
/_licenseprobe (or use an alternative probe) meanwhile?
Diagnostics I can provide on request
-
Recent
journalctl -u elasticsearch(nothing obvious around “license” right now). -
GET /_nodes?filter_path=nodes.*.modules.nameoutput. -
Verbose ES REST handler list, if there’s a way to dump registered handlers.
-
Anything else you need.
Thanks a lot — happy to run any extra checks.