Since updating to 7.1.1 I have been unable to get data to Elasticsearch consistently. I posted about 429 errors the other day which seem to have gone away, but now I have no data coming into ES through Logstash. (Filebeats is still sending some stuff to ES directly.) I DO see the following line in the logs which alarms me:
[2019-06-12T14:01:27,944][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch bulk_path=>"/_monitoring/bulk?system_id=logstash&system_api_version=7&interval=1s", hosts=>[http://10.226.1.94:9200/], sniffing=>false, manage_template=>false, id=>"70ec63d7c8ab7751c5ebfec8e9feab2b65659a7367c48d4dd5573acf1fe8eb6c", document_type=>"%{[@metadata][document_type]}", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_75350e7e-f0cc-4b10-8f44-f8f24e92652f", enable_metric=>true, charset=>"UTF-8">, workers=>1, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, ilm_enabled=>"auto", ilm_rollover_alias=>"logstash", ilm_pattern=>"{now/d}-000001", ilm_policy=>"logstash-policy", action=>"index", ssl_certificate_verification=>true, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
However, my elasticsearch output is configured very simply:
output {
elasticsearch { hosts => ["10.226.1.95:9200", "10.226.1.92:9200", "10.226.1.94:9200"] }
}
I don't know where else it could be picking up document_type from, or even if that might be what's stopping me. any assistance would be appreciated, and thank you in advance!