Can't get logstash to work with Elastic 8.01

[kim_frederiksen]

Hi Everybody,

I am extremely new to ELK and have yet to build the first real solution on the platform. We have had version 7.16.1 and we could see the logs from filebeat in kibana. But before we really begin building stuff we would upgrade to version 8.01. But we simply can't get logstash to work.
We are using the logstash-sample.conf "as is". But we are getting this error:
'[2022-03-04T11:58:48,711][INFO ][logstash.outputs.Elasticsearch][main] Failed to perform request {:message=>"localhost:9200 failed to respond", :exception=>Manticore::ClientProtocolException, :cause=>org.apache.http.NoHttpResponseException: localhost:9200 failed to respond}
[2022-03-04T11:58:48,712][WARN ][logstash.outputs.Elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::ClientProtocolException] localhost:9200 failed to respond"}'

Both Elasticsearch and Kibana seems to be working. So what is the issue here?

Update - If I disable xpack in the Elasticsearch.yml it works. But how do I get it to work together with xpack enabled?

Best regards
Kim

[intrepid1]

Hi there,

There is not really much to go on.

Its not clear whether Logstash is a part of your 7.16.1 set-up or whether it being introduced as a part of your upgrade.

If you can ship data to Elasticsearch via Filebeat, then there should be no reason why you shouldn't be able to connect via Logstash. Take a look at the output section in your filebeat.yml and make sure you all your connection details are correct.

By the looks of the error, it appears that you are using http and not https. How are you connecting from Filebeat?

[kim_frederiksen]

I think what I am trying to say is that things worked in 7.16.1 and after upgrading to 8.01 we can't get logstash to connect to Elasticsearch using the same config files all the way around. Xpack is surely to blaim. The logstash config file used to deliver to http://localhost:9200, but that url is not active when using xpack as it uses https. We could just go back to use version 7.16.1. But at some point we will have to upgrade and have the same issue then.

[intrepid1]

I assume that both Elasticsearch and Logstash are version 8? What exactly is the error message? It would be useful. An obfuscated output section would also be useful.