Data not going from Logstash to Elasticsearch (7.1.1)

Since updating to 7.1.1 I have been unable to get data to Elasticsearch consistently. I posted about 429 errors the other day which seem to have gone away, but now I have no data coming into ES through Logstash. (Filebeats is still sending some stuff to ES directly.) I DO see the following line in the logs which alarms me:

[2019-06-12T14:01:27,944][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch bulk_path=>"/_monitoring/bulk?system_id=logstash&system_api_version=7&interval=1s", hosts=>[], sniffing=>false, manage_template=>false, id=>"70ec63d7c8ab7751c5ebfec8e9feab2b65659a7367c48d4dd5573acf1fe8eb6c", document_type=>"%{[@metadata][document_type]}", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_75350e7e-f0cc-4b10-8f44-f8f24e92652f", enable_metric=>true, charset=>"UTF-8">, workers=>1, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, ilm_enabled=>"auto", ilm_rollover_alias=>"logstash", ilm_pattern=>"{now/d}-000001", ilm_policy=>"logstash-policy", action=>"index", ssl_certificate_verification=>true, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}

However, my elasticsearch output is configured very simply:

output {
 elasticsearch { hosts => ["", "", ""] }

I don't know where else it could be picking up document_type from, or even if that might be what's stopping me. any assistance would be appreciated, and thank you in advance!

If you have x-pack monitoring enabled then you will get this. It would be up to the elastic team fix it. There are multiple issues open for it. This one , for example.

While that's good news in as much as it's not just me missing something obvious, I'm still short a rationale explanation why I'm not seeing data in ES from logstash. I'll try to find other errors that might be more elucidating, thanks for the reply!


OK, when I first noticed the problem, one of the first things I did was to restart the logstash server. Since then, I've been just restarting the logstash service. On a lark, and grasping for straws, I decided to go ahead and restart the entire server again.

Wouldn't you know, ES is getting data again. sigh

Anyrate, just thought I'd share. Thank you again, Badger, for your quick response earlier!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.