I've upgraded to 7.0 today and I've been seeing this for a while now in my logstash log. Can someone please tell me what I should do to fix this?
[
2019-04-15T10:37:11,050][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::Elasticsearch bulk_path=>"/_monitoring/bulk?system_id=logstash&system_api_version=7&interval=1s", hosts=>[http://localhost:9200], sniffing=>false, manage_template=>false, id=>"e466e6b1b36159c3e6c853f5e542e7acc3ca4725ff961494f5c31cbca14350f9", document_type=>"%{[@metadata][document_type]}", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_ac2c5cc6-e683-47ec-a3dc-c77bb2a22b19", enable_metric=>true, charset=>"UTF-8">, workers=>1, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, ilm_enabled=>"auto", ilm_rollover_alias=>"logstash", ilm_pattern=>"{now/d}-000001", ilm_policy=>"logstash-policy", action=>"index", ssl_certificate_verification=>true, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
I have also have this fill up my log but I think I resolved it by deleting the template and using the 7.x json and I don't see any errors related to it
[2019-04-15T10:22:57,429][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"netflow-2019.04.15", :_type=>"_doc", :routing=>nil}, #LogStash::Event:0x20784b76], :response=>{"index"=>{"_index"=>"netflow-2019.04.15", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"The [default] mapping cannot be updated on index [netflow-2019.04.15]: defaults mappings are not useful anymore now that indices can have at most one type."}}}}
elasticsearch log after the change
[2019-04-15T10:32:16,527][INFO ][o.e.c.m.MetaDataIndexTemplateService] [spelk] removing template [netflow]
[2019-04-15T10:35:42,116][INFO ][o.e.c.m.MetaDataIndexTemplateService] [spelk] adding template [netflow] for index patterns [netflow-*]
[2019-04-15T10:37:17,087][INFO ][o.e.c.m.MetaDataCreateIndexService] [spelk] [netflow-2019.04.15] creating index, cause [auto(bulk api)], templates [default, netflow], shards [1]/[0], mappings [_doc]
[2019-04-15T10:37:17,223][INFO ][o.e.c.r.a.AllocationService] [spelk] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[netflow-2019.04.15][0]] ...]).
[2019-04-15T10:37:17,262][INFO ][o.e.c.m.MetaDataMappingService] [spelk] [netflow-2019.04.15/CDP-egW_TTSegG1e7RY7DA] update_mapping [_doc]
My output section of logstash conf, should I specify index => "logstash-%{+YYYY.MM.dd}" }
output {
TEST will send ALL logs to...
#stdout { codec => rubydebug }
#file { path => "/home/user/debug.log" }
#}if [type] == "netflow" {
elasticsearch { hosts => ["http://localhost:9200"] #}
index => "netflow-%{+YYYY.MM.dd}" }
#stdout { codec => rubydebug }
#file { path => "/loggy/netflow.log" }
}else if [type] == "syslog" {
elasticsearch { hosts => ["http://localhost:9200"] }
#stdout { codec => rubydebug }
#file { path => "/loggy/syslog.log" codec => json_lines }
}else if [type] == "wineventlog" {
elasticsearch {
hosts => ["http://localhost:9200"]
#index => "winlogbeat-%{+YYYY.MM.dd}"
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
##document_type => "%{[@metadata][type]}"
}
#stdout { codec => rubydebug }
#file { path => "/loggy/winlogbeat.log" }
}else if [type] == "vcenter" {
elasticsearch { hosts => ["http://localhost:9200"] }
#file { path => "/loggy/vcenter.log"}
}else if [type] == "cisco-asa" {
elasticsearch { hosts=> ["http://localhost:9200"] #}
#Remove index to ingest using logstash
index => "cisco-asa-%{+YYYY.MM.dd}" }
#stdout { codec => rubydebug }
#file { path => "/loggy/cisco-asa.log" }
}else if [type] == "netflow-wlc" {
elasticsearch { hosts => ["http://localhost:9200"] }
#stdout { codec => rubydebug }
#file { path => "/loggy/netflow-wlc.log" }
}else {
file { path=> "/loggy/unknown.log" codec => json_lines }
}
}