Logstash breaks when disabling certain X-pack features

When installing x-pack on a fresh Ubuntu 17.04 install, (Logstash 5.4.1, Elasticsearch 5.4.1, Kibana 5.4.1 from Elastic repo), Logstash breaks when disabling monitoring (xpack.monitoring.enabled: false in elasticsearch.yml and kibana.yml).

Using unmodified packages from Elastic, Logstash fails with the following error message:

[2017-06-14T21:45:40,102][ERROR][logstash.outputs.elasticsearch] Encountered an unexpected error submitting a bulk request! Will retry. {:error_message=>"uninitialized constant LogStash::Outputs::ElasticSearch::Common::RETRYABLE_CODES", :class=>"NameError", :backtrace=>["org/jruby/RubyModule.java:2719:in 'const_missing'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.1-java/lib/logstash/outputs/elasticsearch/common.rb:248:in 'safe_bulk'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.1-java/lib/logstash/outputs/elasticsearch/common.rb:118:in 'submit'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.1-java/lib/logstash/outputs/elasticsearch/common.rb:86:in 'retrying_submit'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.1-java/lib/logstash/outputs/elasticsearch/common.rb:37:in 'multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:in 'multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:47:in 'multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:407:in 'output_batch'", "org/jruby/RubyHash.java:1342:in 'each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:406:in 'output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:352:in 'worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:317:in `start_workers'"]}

Then I upgraded the logstash-output-elasticsearch plugin:

# /usr/share/logstash/bin/logstash-plugin update logstash-output-elasticsearch
Updating logstash-output-elasticsearch
Updated logstash-output-elasticsearch 7.3.1 to 7.3.5

Logstash still fails, but with a new error message:

[2017-06-14T22:35:59,087][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>400, :response_body=>"{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"request [/_xpack/monitoring/_bulk] contains unrecognized parameters: [interval], [system_api_version], [system_id]"}],"type":"illegal_argument_exception","reason":"request [/_xpack/monitoring/_bulk] contains unrecognized parameters: [interval], [system_api_version], [system_id]"},"status":400}"}

Enabling monitoring (removing the settings from elasticsearch.yml and kibana.yml) makes the errors go away and Logstash works as expected.

Issue is consistent and has been reproduced twice, with Ubuntu 16.10 and 17.04.

Have you tried setting xpack.monitoring.enabled: false in your logstash.yml?

Thank you, that made the errors go away and Logstash works as expected.

The X-pack documentation at https://www.elastic.co/guide/en/x-pack/current/monitoring-settings.html should be updated to reflect this requirement. Currently it states that xpack.monitoring.enabled: false should be added to elasticsearch.yml and kibana.yml only.

On a related note, the X-pack settings start page (https://www.elastic.co/guide/en/x-pack/current/xpack-settings.html) also specifies elasticsearch.yml and kibana.yml only.

That's a good point, definitely an oversight. Thanks for the heads up, I've opened an issue get it fixed.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.