# ERROR Connection marked as failed because the onConnect callback failed: could not connect to a compatible version of Elasticsearch: 404 Not Found: **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [October 22, 2025, 1:45pm UTC](https://discuss.elastic.co/t/error-connection-marked-as-failed-because-the-onconnect-callback-failed-could-not-connect-to-a-compatible-version-of-elasticsearch-404-not-found/382888 "2025-10-22T13:45:50Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Rovsen\_Agayev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rovsen_agayev/32/139907_2.png) [@Rovsen\_Agayev](https://discuss.elastic.co/u/Rovsen_Agayev)\ **Post date:** [October 22, 2025, 1:45pm UTC](https://discuss.elastic.co/t/error-connection-marked-as-failed-because-the-onconnect-callback-failed-could-not-connect-to-a-compatible-version-of-elasticsearch-404-not-found/382888/1 "2025-10-22T13:45:50Z")
## Filebeat 9.1.5 → Elasticsearch 9.1.5: `/_license?human=false` returns **404 {}** , Filebeat fails with “onConnect callback failed: 404 {}” ### Environment - **Elasticsearch:** 9.1.5 (Deb package, `build_flavor: default`) - **Kibana:** 9.1.5 - **Filebeat:** 9.1.5 - **Cluster:** 3 nodes, TLS enabled (auto-generated `http_ca.crt`), security enabled - **OS:** Linux (Debian/Ubuntu family) - **Proxy / LB in front of ES:** **No** (direct connection) ### Symptom Filebeat connects to ES but immediately fails during the compatibility check: ```auto Failed to connect to backoff(elasticsearch(https://10.0.100.61:9200)): Connection marked as failed because the onConnect callback failed: could not connect to a compatible version of Elasticsearch: 404 Not Found: { } ``` With debug: ```auto ... esclientleg.(*Connection).Request GET https://10.0.100.61:9200/_license?human=false → 404 {} ``` ### What I verified **1) ES root endpoint OK** ```auto curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://10.0.100.61:9200/ # 200 OK, shows version 9.1.5 (default) ``` **2) The exact endpoint Filebeat calls returns 404** ```auto curl -i --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic \ 'https://10.0.100.61:9200/_license?human=false' HTTP/1.1 404 Not Found X-elastic-product: Elasticsearch content-type: application/json content-length: 4 { } ``` (Same result on `https://127.0.0.1:9200/_license?human=false` locally on the node.) **3) Cluster is healthy** ```auto curl -k -u elastic https://10.0.100.61:9200/_cluster/health?pretty # status: green ``` **4) X-Pack modules are present (this is the default distribution)** ```auto ls -1 /usr/share/elasticsearch/modules | grep -i x-pack | wc -l # many x-pack-* modules present curl -s --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic \ "https://10.0.100.61:9200/_cat/plugins?v" # empty (understandable: x-pack is a module, not a plugin) ``` **5) No custom filtering for HTTP paths** - There is **no** `http.filter.path` or similar in `elasticsearch.yml`. - Nothing sits in front of ES (no nginx/haproxy). **6) Node/cluster license settings** ```auto curl -s --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic \ "https://127.0.0.1:9200/_nodes/settings?filter_path=nodes.*.settings.xpack.license" # {} curl -s --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic \ "https://127.0.0.1:9200/_cluster/settings?include_defaults=true&filter_path=defaults.xpack.license,persistent.xpack.license,transient.xpack.license" | jq . # shows defaults with self_generated.type=basic ``` **7) Filebeat test output (TLS + fingerprint)** ```auto filebeat test output \ -E output.elasticsearch.hosts=["https://10.0.100.61:9200"] \ -E output.elasticsearch.username=elastic \ -E output.elasticsearch.password=' *****' \ -E output.elasticsearch.ssl.ca_trusted_fingerprint='598D8B23...F481' # TLS handshake OK, Ping 200, then GET /_license?human=false → 404 {} ``` ### Config snippets (sanitized) **/etc/filebeat/filebeat.yml** ```auto setup.kibana: host: "http://10.0.100.61:5601" output.elasticsearch: hosts: ["https://10.0.100.61:9200"] preset: balanced ssl.ca_trusted_fingerprint: " *****" username: "elastic" password: " *****" filebeat.inputs: - type: syslog enabled: true protocol.udp: host: "0.0.0.0:5514" tags: ["mikrotik","syslog"] ``` **/etc/elasticsearch/elasticsearch.yml** ```auto cluster.name: Firewall node.name: node-1 node.roles: [master, data_hot, ingest, data_content] path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch network.host: 0.0.0.0 http.port: 9200 transport.port: 9300 discovery.seed_hosts: ["10.0.100.61:9300","10.0.100.62:9300","10.0.100.63:9300"] xpack.security.enabled: true xpack.security.enrollment.enabled: true xpack.security.http.ssl: enabled: true keystore.path: certs/http.p12 xpack.security.transport.ssl: enabled: true verification_mode: certificate keystore.path: certs/transport.p12 truststore.path: certs/transport.p12 cluster.initial_master_nodes: ["node-1","node-2","node-3"] http.host: 0.0.0.0 ``` ### Timeline / extra context - This setup **was working** earlier and then suddenly stopped working for Filebeat outputs. - Fleet-Server/Agent are healthy and ES/Kibana work normally. Only Filebeat’s ES output fails because `/_license?human=false` returns 404 `{}`. ### Questions for the Elastic team 1. In Elasticsearch **9.1.5 (default flavor)**, should `GET /_license?human=false` always be available? Under what conditions would it return **404** (instead of 200/401/403)? 2. Is there any setting (cluster or node) that can disable the license route without logging an error/warning? I don’t have `http.filter.path` and there is no reverse proxy. 3. Could this be a regression/bug in 9.1.5 where the license endpoint is not registered in some startup order? If so, which logs/diagnostics can I capture to help? 4. Any suggested workaround to let Filebeat skip the `/_license` probe (or use an alternative probe) meanwhile? ### Diagnostics I can provide on request - Recent `journalctl -u elasticsearch` (nothing obvious around “license” right now). - `GET /_nodes?filter_path=nodes.*.modules.name` output. - Verbose ES REST handler list, if there’s a way to dump registered handlers. - Anything else you need. Thanks a lot — happy to run any extra checks. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 22, 2025, 1:58pm UTC](https://discuss.elastic.co/t/error-connection-marked-as-failed-because-the-onconnect-callback-failed-could-not-connect-to-a-compatible-version-of-elasticsearch-404-not-found/382888/2 "2025-10-22T13:58:57Z")
Hi @Rovsen_Agayev, Welcome to the community... > [@Rovsen\_Agayev](#): > > ```auto > ... esclientleg.(*Connection).Request GET https://10.0.100.61:9200/_license?human=false > → 404 {} > > ``` Unfortunately, sometimes the error messages can be confusing that 404 may actually be a connectivity issue. > [@Rovsen\_Agayev](#): > > `/etc/elasticsearch/certs/http_ca.crt ` > [@Rovsen\_Agayev](#): > > ```auto > output.elasticsearch: > hosts: ["https://10.0.100.61:9200"] > preset: balanced > ssl.ca_trusted_fingerprint: " *****" > username: "elastic" > password: " *****" > > ``` For Debug can you try this and report back. Not for production but a test. All your test that work use the CA those that dont use the fingerprint do not. ```auto output.elasticsearch: hosts: ["https://10.0.100.61:9200"] preset: balanced ssl.verification_mode: none username: "elastic" password: " *****" ``` ---
**Author:** ![Rovsen\_Agayev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rovsen_agayev/32/139907_2.png) [@Rovsen\_Agayev](https://discuss.elastic.co/u/Rovsen_Agayev)\ **Post date:** [October 22, 2025, 3:31pm UTC](https://discuss.elastic.co/t/error-connection-marked-as-failed-because-the-onconnect-callback-failed-could-not-connect-to-a-compatible-version-of-elasticsearch-404-not-found/382888/3 "2025-10-22T15:31:13Z")
Same root@elk:~# filebeat test config Config OK root@elk:~# filebeat test output elasticsearch: htt ps://10.0.100.61:9200... parse url... OK connection... parse host... OK dns lookup... OK addresses: 10.0.100.61 dial up... OK TLS... security... WARN server's certificate chain verification is disabled handshake... OK TLS version: TLSv1.3 dial up... OK talk to server... ERROR Connection marked as failed because the onConnect callback failed: could not connect to a compatible version of Elasticsearch: 404 Not Found: { } ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 22, 2025, 4:10pm UTC](https://discuss.elastic.co/t/error-connection-marked-as-failed-because-the-onconnect-callback-failed-could-not-connect-to-a-compatible-version-of-elasticsearch-404-not-found/382888/4 "2025-10-22T16:10:45Z")
Brand new 9.1.5 Have you updated / deleted / changed the license? Everything seems to work as expected `curl` shows proper responses `filebeat` test output works ```auto $ curl -u elastic http://localhost:9200 Enter host password for user 'elastic': { "name" : "f4ad43d10d71", "cluster_name" : "docker-cluster", "cluster_uuid" : "C2-S2hw3Tpq0QCETjYav0A", "version" : { "number" : "9.1.5", "build_flavor" : "default", "build_type" : "docker", "build_hash" : "90ee222e7e0136dd8ddbb34015538f3a00c129b7", "build_date" : "2025-10-02T22:07:12.966975992Z", "build_snapshot" : false, "lucene_version" : "10.2.2", "minimum_wire_compatibility_version" : "8.19.0", "minimum_index_compatibility_version" : "8.0.0" }, "tagline" : "You Know, for Search" } $ curl -u elastic http://localhost:9200/_license?human=false Enter host password for user 'elastic': { "license" : { "status" : "active", "uid" : "69862c9d-19a9-46e6-93ad-473ffa88b72b", "type" : "trial", "issue_date" : "2025-10-22T16:03:56.559Z", "issue_date_in_millis" : 1761149036559, "expiry_date" : "2025-11-21T16:03:56.559Z", "expiry_date_in_millis" : 1763741036559, "max_nodes" : 1000, "max_resource_units" : null, "issued_to" : "docker-cluster", "issuer" : "elasticsearch", "start_date_in_millis" : -1 } } ``` Filebeat No need to define everything on the command line... ```auto filebeat-9.1.5-darwin-aarch64 ./filebeat test output elasticsearch: http://localhost:9200... parse url... OK connection... parse host... OK dns lookup... OK addresses: ::1, 127.0.0.1 dial up... OK TLS... WARN secure connection disabled talk to server... OK version: 9.1.5 ``` ```auto curl -u elastic \ "http://localhost:9200/_nodes/settings?filter_path=nodes.*.settings.xpack.license" Enter host password for user 'elastic': {"nodes":{"dmWH6Kv0QcC_IAdYEwykSQ":{"settings":{"xpack":{"license":{"self_generated":{"type":"trial"}}}}}}}% ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 22, 2025, 4:24pm UTC](https://discuss.elastic.co/t/error-connection-marked-as-failed-because-the-onconnect-callback-failed-could-not-connect-to-a-compatible-version-of-elasticsearch-404-not-found/382888/5 "2025-10-22T16:24:33Z")
CORRECTED Apologies...Just saw this... Thinking your routes are not working correctly > [@Rovsen\_Agayev](#): > > **Proxy / LB in front of ES:** **No** (direct connection) Correction sorry I read that wrong. I thought it said you DID have a load balancer rereading, I see that you do not. Here are some direct answers to your questions below > 1. In Elasticsearch **9.1.5 (default flavor)**, should `GET /_license?human=false` always be available? Under what conditions would it return **404** (instead of 200/401/403)? No should be available, especially with "vanilla" setup > 1. Is there any setting (cluster or node) that can disable the license route without logging an error/warning? I don’t have `http.filter.path` and there is no reverse proxy. Not that I know of > 1. Could this be a regression/bug in 9.1.5 where the license endpoint is not registered in some startup order? If so, which logs/diagnostics can I capture to help? Not likely, as I just setup entire 9.1.5 stack everything connected , responding / reported correctly. You can enable DEBUG logging in elasticsearch to see what is going on > 1. Any suggested workaround to let Filebeat skip the `/_license` probe (or use an alternative probe) meanwhile? No... There is something else basic going on... As I just stood up a whole 9.1.5 and ran all the test and everything looks good. ---
**Author:** ![Rovsen\_Agayev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rovsen_agayev/32/139907_2.png) [@Rovsen\_Agayev](https://discuss.elastic.co/u/Rovsen_Agayev)\ **Post date:** [October 23, 2025, 5:32am UTC](https://discuss.elastic.co/t/error-connection-marked-as-failed-because-the-onconnect-callback-failed-could-not-connect-to-a-compatible-version-of-elasticsearch-404-not-found/382888/6 "2025-10-23T05:32:15Z")
Hi Stephen, Thanks for your reply. I’ve tried exactly what you described — the same version (Elasticsearch and Filebeat 9.1.5), direct connection, and no load balancer or proxy in between. When I test manually with `curl -u elastic ``https://10.0.100.61:9200/_license?human=false`, it returns `200 OK` and shows the correct license info. However, Filebeat still gets: `could not connect to a compatible version of Elasticsearch: 404 Not Found: { }` One thing I noticed is that my cluster is running an **Enterprise license** (not trial). Do you think that could make a difference here? Everything else (Kibana, curl tests, etc.) works fine — only Filebeat fails on the `_license` check. Appreciate any suggestions on what else I can check or capture to help debug this further. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [October 23, 2025, 6:23am UTC](https://discuss.elastic.co/t/error-connection-marked-as-failed-because-the-onconnect-callback-failed-could-not-connect-to-a-compatible-version-of-elasticsearch-404-not-found/382888/7 "2025-10-23T06:23:26Z")
Well first you have an Enterprise license you should open a support ticket. No I didn't think an enterprise license would cause this issue. > [@Rovsen\_Agayev](#): > > When I test manually with `curl -u elastic ``https://10.0.100.61:9200/_license?human=false`, it returns `200 OK` and shows the correct license info. That is different than you reported before, what changed? And you are trying the `curl` from the filebeat host?? And your not running like the OSS version of fiiebeat or something? (not sure if that makes a difference)