Logstash 6.3 requires x-pack security?

Elastic Stack Logstash
1

Hey everyone,

So I just upgraded our ES cluster to version 6.3. All nodes are up and running, as well as Kibana. Unfortunately our logstash instances refuse to connect to elasticsearch now, claiming that X-Pack Security needs to be enabled in Elasticsearch:

[2018-06-18T16:19:34,240][INFO ][logstash.configmanagement.bootstrapcheck] Using Elasticsearch as config store {:pipeline_id=>["oven", "paste_printing", "paste_inspection", "metricbeats", "winlogbeats", "aoi", "laser", "alt_asm_xml", "alt_xry_data"], :poll_interval=>"5000000000ns"}
[2018-06-18T16:19:34,441][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://logstash_system:xxxxxx@XXXXXXXX:9200/]}}
[2018-06-18T16:19:34,443][INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@XXXXXX:9200/, :path=>"/"}
[2018-06-18T16:19:34,589][WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"http://logstash_system:xxxxxx@XXXXXXX:9200/"}
[2018-06-18T16:19:34,617][INFO ][logstash.licensechecker.licensereader] ES Output version determined {:es_version=>6}[2018-06-18T16:19:34,618][WARN ][logstash.licensechecker.licensereader] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-06-18T16:19:34,639][ERROR][logstash.configmanagement.elasticsearchsource] X-Pack Security needs to be enabled in Elasticsearch. Please set xpack.security.enabled: true in elasticsearch.yml.
[2018-06-18T16:19:34,643][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::LicenseChecker::LicenseError: X-Pack Security needs to be enabled in Elasticsearch. Please set xpack.security.enabled: true in elasticsearch.yml.>, :backtrace=>["/usr/share/logstash/x-pack/lib/license_checker/licensed.rb:67:in `with_license_check'", "/usr/share/logstash/x-pack/lib/config_management/elasticsearch_source.rb:47:in `initialize'", "/usr/share/logstash/x-pack/lib/config_management/hooks.rb:42:in `after_bootstrap_checks'", "/usr/share/logstash/logstash-core/lib/logstash/event_dispatcher.rb:34:in `block in fire'", "/usr/share/logstash/logstash-core/lib/logstash/event_dispatcher.rb:32:in `fire'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:294:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:238:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `<main>'"]}
[2018-06-18T16:19:34,649][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit`

The configuration file for logstash has the following xpack configuration (IP addresses omitted):

xpack.management.enabled: true
xpack.management.elasticsearch.url: "http://XXXXXXXX:9200/"
xpack.management.logstash.poll_interval: 5s
xpack.management.pipeline.id: ["oven", "paste_printing", "paste_inspection", "metricbeats", "winlogbeats","aoi","laser","alt_asm_xml","alt_xry_data"]
xpack.management.elasticsearch.password: elasticsearch
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: "http://XXXXXXX:9200/"
xpack.monitoring.elasticsearch.password: elasticsearch

In the logstash configuraiton file we've had the xpack.security.enabled flag set to false thus far.

I checked the release notes and breaking changes, etc... but couldn't find anything.

Is there still a way to run logstash w/o security enabled or will have to setup security now?

2

No, security is not required. I would start with removing the xpack.management.elasticsearch.password and xpack.monitoring.elasticsearch.password settings as these are not applicable without security enabled and see if that makes a difference.

3

I already tried that. I then got the following error:

You must set the password using the "xpack.management.elasticsearch.password" in logstash.yml

See here:

[2018-06-18T17:04:39,030][INFO ][logstash.configmanagement.bootstrapcheck] Using Elasticsearch as     config store {:pipeline_id=>["oven", "paste_printing", "paste_inspection", "metricbeats", "winlogbeats", "aoi", "laser", "alt_asm_xml", "alt_xry_data"], :poll_interval=>"5000000000ns"}
[2018-06-18T17:04:39,039][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<ArgumentError: You must set the password using the "xpack.management.elasticsearch.password" in logstash.yml>, :backtrace=>["/usr/share/logstash/x-pack/lib/config_management/elasticsearch_source.rb:40:in `initialize'", "/usr/share/logstash/x-pack/lib/config_management/hooks.rb:42:in `after_bootstrap_checks'", "/usr/share/logstash/logstash-core/lib/logstash/event_dispatcher.rb:34:in `block in fire'", "/usr/share/logstash/logstash-core/lib/logstash/event_dispatcher.rb:32:in `fire'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:294:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:238:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `<main>'"]}
[2018-06-18T17:04:39,044][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

This has actually been the case for the past few versions, that we had to set the password even if security was turned off.

4

I am not a ruby coder, but searching the repo on Github I found the following in the read_configuration_spec.rb file:

context "security is disabled" do
    before(:all) do
      elasticsearch_options = {
        :settings => {
          "xpack.security.enabled" => false
        }
      }
      start_services(elasticsearch_options, logstash_options(PIPELINE_ID, /X-Pack Security needs to be enabled in Elasticsearch/))
    end

    after(:all) do
      stop_services
    end

    it "fails to start" do
      temporary_file = File.join(Stud::Temporary.directory, "hello.log")
      new_config = "input { generator { count => 10000 }} output { file { path => '#{temporary_file}' } }"
      expect(File.exist?(temporary_file)).to be_falsey
      push_elasticsearch_config(PIPELINE_ID, new_config)
      elasticsearch_client.indices.refresh

      Stud.try(MAX_RETRY.times, [RSpec::Expectations::ExpectationNotMetError]) do
        expect(File.exist?(temporary_file)).to be_falsey
      end
    end

Here the link to the file: https://github.com/elastic/logstash/blob/93cad10da1469efa3b72b0a4058353e020d25dda/x-pack/qa/integration/management/read_configuration_spec.rb

Is it possible that during the x-pack integration this changed, and the security check is now forced?

5

I do not have security enabled. I have the following in my configuration for reasons I no longer recall. Perhaps it helps?

xpack.management.elasticsearch.username: ""
xpack.management.elasticsearch.password: ""
6

It's listed under known issues:

Known issues
Clusters with a Gold or Platinum license that are upgrading to 6.3 will need to explicitly set xpack.security.enabled: true in elasticsearch.yml to upgrade successfully. If this value is not set, the cluster will be unable to form after upgrade.

Also see Breaking changes in 6.3.

Unfortunately nothing is noted in the breaking changes, or in regard to logstash having this requirement for 6.3. I tried with empty username and password with no avail. Guess we'll have to enable security...

It'd be helpful if this was indeed listed as a breaking change, or under the release notes for elasticsearch, or logstash. Neither of these places listed this issue!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.