Undefined method `sanitized'

Elastic Stack Logstash
1

Hi there,
I upgraded ELK from 5.5. to 5.6, and logstash received the log, parsed, but didn't write to elasticsearch. From log, it complains ""undefined method `sanitized' ", logstash-output-elasticsearch version is 7.4.0

Please help.

Thanks and Regards,
Fei

[2017-09-14T14:30:11,367][DEBUG][logstash.pipeline ] output received {"event"=>{"@timestamp"=>2017-09-14T18:30:05.000Z, "offset"=>9574607, "apache2"=>{"access"=>{"referrer"=>"-", "response_code"=>"200", "remote_ip"=>"13.115.21.9", "geoip"=>{"timezone"=>"America/Toronto", "ip"=>"13.115.21.9", "latitude"=>43.7334, "continent_code"=>"NA", "city_name"=>"Toronto", "country_name"=>"Canada", "country_code2"=>"CA", "country_code3"=>"CA", "region_name"=>"Ontario", "location"=>{"lon"=>-79.3329, "lat"=>43.7334}, "postal_code"=>"m3c 3n3", "region_code"=>"ON", "longitude"=>-79.3329}, "method"=>"LOCK", "user_name"=>"ykozhem", "http_version"=>"1.1", "body_sent"=>{"bytes"=>"378"}, "url"=>"/csis/trunk/ddl_applications/CSISPROD/Functions/get_future_date.sql", "user_agent"=>{"name"=>"Other", "os_name"=>"Other", "os"=>"Other", "device"=>"Other", "build"=>""}}}, "@version"=>"1", "input_type"=>"log", "beat"=>{"name"=>"jenkins", "hostname"=>"jenkins", "version"=>"5.5.2"}, "host"=>"jenkins", "read_timestamp"=>"2017-09-14T18:30:08.222Z", "source"=>"/var/log/httpd/access_log", "fields"=>{"document_type"=>"svn_access_log"}, "type"=>"log", "tags"=>["beats_input_codec_plain_applied"]}}
[2017-09-14T14:30:14,092][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-09-14T14:30:15,742][ERROR][logstash.outputs.elasticsearch] Encountered an unexpected error submitting a bulk request! Will retry. {:error_message=>"undefined method sanitized' for #<String:0x3145e6f7>", :class=>"NoMethodError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:249:insafe_bulk'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:222:in safe_bulk'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:119:insubmit'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:87:in retrying_submit'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:38:inmulti_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:49:inmulti_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:436:in output_batch'", "org/jruby/RubyHash.java:1342:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:435:in output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:381:inworker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:342:in `start_workers'"]}

2

I started getting that error also today after I installed x-pack. I removed and that seems to have fixed the problem.

I guess I'll look into x-pack more tomorrow.

3

I updated all Logstash plugins, logstash-output-elasticsearch (8.1.1)

now got below error:
undefined method `body'

[2017-09-15T07:39:08,104][ERROR][logstash.outputs.elasticsearch] Encountered an unexpected error submitting a bulk request! Will retry. {:error_message=>"undefined method body' for #<LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError:0x7bf1351>", :class=>"NoMethodError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-8.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:251:insafe_bulk'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-8.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:223:in safe_bulk'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-8.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:113:insubmit'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-8.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:81:in retrying_submit'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-8.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:33:inmulti_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:49:inmulti_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:436:in output_batch'", "org/jruby/RubyHash.java:1342:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:435:in output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:381:inworker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:342:in `start_workers'"]}

4

After removed x-pack, got infinite loop like below:

[2017-09-15T07:36:27,607][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x30f9a63b @kibana_version_parts=["5", "6", "0"], @module_name="fb_apache", @directory="/usr/share/logstash/modules/fb_apache/configuration">}
[2017-09-15T07:36:27,608][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}

5

This error is due to a security permissions problem with the user configured in the output. The error message displayed is obviously a mistake, and has been fixed in version 7.4.2 of the output plugin:

$ bin/logstash-plugin update logstash-output-elasticsearch
Updating logstash-output-elasticsearch
Updated logstash-output-elasticsearch 7.4.1 to 7.4.2

Now you should receive a sane error,

[2017-09-22T10:43:56,826][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>403, :url=>"http://127.0.0.1:9200/_bulk"}

, which indicates that the user does not have permission to index data. In order to create a dedicated logstash write user, see this guide, or use the elastic superuser while testing.

6

Thanks, I will try it and update you.

7

I'm getting this error after a clean install of 5.6.1. I updated the plugins and now I'm getting the issue I posted about here. Logstash 5.6.1 inifinite loop? Encountered a retryable error

@PhaedrusTheGreek any help would be appreciated.

8

I could not find the solution but have a workaround for this. I have fresh installation of logstash 5.6.3 and using logstash-output-elasticsearch plugin version 9.0.0(upgraded from 7.4.2)

Error:
[2017-10-11T06:52:46,492][ERROR][logstash.outputs.elasticsearch] Encountered an unexpected error submitting a bulk request! Will retry. {:error_message=>"undefined method sanitized' for \"/_bulk\":String", :cl ass=>"NoMethodError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-9.0.0-java/lib/logstash/outputs/elasticsearch/common.rb:250:insafe_bulk'", "/usr/share/logsta
sh/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-9.0.0-java/lib/logstash/outputs/elasticsearch/common.rb:113:in submit'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elastics earch-9.0.0-java/lib/logstash/outputs/elasticsearch/common.rb:81:inretrying_submit'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-9.0.0-java/lib/logstash/outputs/elasticsea
rch/common.rb:33:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:inmulti_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegat
or.rb:49:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:434:inoutput_batch'", "org/jruby/RubyHash.java:1342:in each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeli ne.rb:433:inoutput_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:381:in worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:342:instart_workers'"]}

This occurs during error handling and url sanitisation method in mentioned ruby file common.rb line 250:
Changed
log_hash = {:code => e.response_code, :url => e.url.sanitized.to_s}
to
log_hash = {:code => e.response_code, :url => e.url, :body => e.response_body }

Now that I am not sanitizing url, hence getting below appropriate message in logs:
[2017-10-11T07:14:25,097][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>404, :url=>"/_bulk", :body=>"{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index and [action.auto_create_index] ([prod*,.security,.monitoring*,.watches,.triggered_watches,.watcher-history*]) doesn't match","index_uuid":"na","index":"logstash-2017.10.11"}],"type":"index_not_found_exception","reason":"no such index and [action.auto_create_index] ([prod*,.security,.monitoring*,.watches,.triggered_watches,.watcher-history*]) doesn't match","index_uuid":"na","index":"logstash-2017.10.11"},"status":404}"}

This was the error due to configuration in elasticsearch.yml file which allowed only certain index pattern to be created automatically. Changed configuration in yml and finally index was created.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.