When installing x-pack on a fresh Ubuntu 17.04 install, (Logstash 5.4.1, Elasticsearch 5.4.1, Kibana 5.4.1 from Elastic repo), Logstash breaks when disabling monitoring (xpack.monitoring.enabled: false in elasticsearch.yml and kibana.yml).
Using unmodified packages from Elastic, Logstash fails with the following error message:
[2017-06-14T21:45:40,102][ERROR][logstash.outputs.elasticsearch] Encountered an unexpected error submitting a bulk request! Will retry. {:error_message=>"uninitialized constant LogStash::Outputs::Elasticsearch::Common::RETRYABLE_CODES", :class=>"NameError", :backtrace=>["org/jruby/RubyModule.java:2719:in 'const_missing'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.1-java/lib/logstash/outputs/elasticsearch/common.rb:248:in 'safe_bulk'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.1-java/lib/logstash/outputs/elasticsearch/common.rb:118:in 'submit'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.1-java/lib/logstash/outputs/elasticsearch/common.rb:86:in 'retrying_submit'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.1-java/lib/logstash/outputs/elasticsearch/common.rb:37:in 'multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:in 'multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:47:in 'multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:407:in 'output_batch'", "org/jruby/RubyHash.java:1342:in 'each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:406:in 'output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:352:in 'worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:317:in `start_workers'"]}
Then I upgraded the logstash-output-elasticsearch plugin:
# /usr/share/logstash/bin/logstash-plugin update logstash-output-elasticsearch Updating logstash-output-elasticsearch Updated logstash-output-elasticsearch 7.3.1 to 7.3.5
Logstash still fails, but with a new error message:
[2017-06-14T22:35:59,087][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>400, :response_body=>"{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"request [/_xpack/monitoring/_bulk] contains unrecognized parameters: [interval], [system_api_version], [system_id]"}],"type":"illegal_argument_exception","reason":"request [/_xpack/monitoring/_bulk] contains unrecognized parameters: [interval], [system_api_version], [system_id]"},"status":400}"}
Enabling monitoring (removing the settings from elasticsearch.yml and kibana.yml) makes the errors go away and Logstash works as expected.
Issue is consistent and has been reproduced twice, with Ubuntu 16.10 and 17.04.