Exiting: error connecting to Kibana: fail to get the Kibana version: fail to parse kibana version (): passed version is not semver:

I'm trying to setup an ELK instance to use as a winlogbeat output, and even though both the ELK stack and winlogbeat versions are the exact same, I'm getting this cryptic error:

Exiting: error connecting to Kibana: fail to get the Kibana version: fail to parse kibana version (): passed version is not semver:

I've linked both a full winlogbeat output when running winlogbeat setup -e and my ELK stack env file.

Log: winlogbeat.txt · GitHub
env: .env · GitHub

I generally see this error when the kibana URL provided is a valid address with something listening but it's not Kibana or it's https but kibana is actually not available over https.

Can you try running both

curl -u "user:password" https://192.168.1.103:5601/api/status

curl -u "user:password" http://192.168.1.103:5601/api/status

Replacing user and password with your actual kibana username and password.

And letting me know what the result is of each?

Can you also share your beat yml config?

curl w/ssl (passing --ignore too) returns the standard kibana json response (kibanastatus.json · GitHub)
curl without ssl returns: curl: (52) Empty reply from server

Here the kibana yml config file: kibana.yml · GitHub

Can you share the winlogbeat.yml?

There you go winlogbeat.yml · GitHub

Can you confirm you ran these from the server running Filebeat?

Can you restart the kibana server and try to run Filebeat again?

Can you try running winlogbeat with winlogbeat setup -e -d '*' and share the full output?

Filebeat is not part of my deployment.
There's just kibana + elastic + logstash.

Here's the log winlogbeat.json · GitHub (just the server ip changed cause dhcp)

Did you perform the curl command from the device running winlogbeat? I assume no?

Can you try to access the previous urls in a browser from the device running winlogbeat or use invoke-webrequest from a power shell session?

Yeah I performed it on my windows client, just through WSL, and yes I can access all of them just fine.

The error appears to indicate that winlogbeat is getting some sort of json response from kibana but that it is not finding any value for version in the json response.

Are you fronting kibana with a proxy or is there any other infrastructure between the Windows box and the kibana server?

Can you try setting up another beat on another device and see if you can perform kibana setup from that device?

The whole ELK stack is deployed in a dedicated VM as a docker stack, using GitHub - sherifabdlnaby/elastdocker: 🐳 Elastic Stack (ELK) v8+ on Docker with Compose. Pre-configured out of the box to enable Logging, Metrics, APM, Alerting, ML, and SIEM features. Up with a Single Command., and there's no proxy in-between.

I already tried that days ago and I was getting the same error.

Can you add setup.kibana.protocol: "https" to your winlogbeat config.

When a node is defined as an IP:PORT, the scheme and path are taken from the setup.kibana.protocol and setup.kibana.path config options.

Done, still the same issue.

Though documentation in the config says: # Scheme and port can be left out and will be set to the default (http and 5601), this is why I didn't add setup.kibana.protocol:

Scheme can be left out if you do not specify a port and if are using http.

If you specify a port then I do believe you need to specify the scheme.

Actually that's wrong. The documentation does say "However, if you specify a URL for host, the value of protocol is overridden by whatever scheme you specify in the URL." So your initial configuration was fine.

Im not able to reproduce this with a kibana install following the Elastic installation guide so if you are testing with the kibana scheme set to https in the winlogbeat config it might make sense to file an issue with that elastic quick start project on GitHub to see if they have done anything that would inhibit winlogbeat setting up Kibana.

You could also try to setup an SSL proxy so you can inspect the request winlogbeat is making and the exact contents of the response which might help figuring out what's going on.

You also could just use the cli to export the dashboards and index patterns and manually import them as kibana setup is optional. Winlogbeat command reference | Winlogbeat Reference [8.12] | Elastic

I've opened a bug report to the Github project (Exiting: error connecting to Kibana: fail to get the Kibana version: fail to parse kibana version (): passed version is not semver: · Issue #112 · sherifabdlnaby/elastdocker · GitHub)

I'll try doing that on Monday and update this thread.

I'd prefer to use kibana.

You can still use kibana, all this setup does is a one time import of the default winlogbeat dashboard.

You could also try setting something like logging.verbose: true in the kibana.yml and see if you can get kibana to log the requests from winlogbeat and it's corresponding responses. That exact setting might be deprecated so if you go this route and have trouble let me know and I can go digging.

FATAL Error: [config validation of [logging].verbose]: definition for this key is missing

They error seems to imply this won't work either but can you try

logging:
  root:
    level: debug

From

Done.
winlogbeats is now displaying a different error

{"log.level":"error","@timestamp":"2024-02-16T17:58:53.655+0100","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.handleError","file.name":"instance/beat.go","file.line":1312},"message":"Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://192.168.1.119:5601/api/status fails: status=503. Response: {\"status\":{\"overall\":{\"level\":\"unavailable\"}}}","service.name":"winlogbeat","ecs.version":"1.6.0"}
Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://192.168.1.119:5601/api/status fails: status=503. Response: {"status":{"overall":{"level":"unavailable"}}}

and from the kibana logs I'm getting [2024-02-16T17:00:37.520+00:00][DEBUG][http.server.response] GET /api/status 503 3ms - 46.0B
along with [2024-02-16T17:00:43.283+00:00][DEBUG][elasticsearch.query.data] [ConnectionError]: unable to verify the first certificate

I'll OOO till Monday though

Ok that looks like an issue with kibana not trusting the elasticsearch certificate.