After update to logstash 7.6.0, it won't start

[andre22]

Hi, I updated from 7.5.2 to 7.6.0 (debian stretch).

The log gves me the following error, I struggle to see where this comes from:

[2020-02-11T21:30:28,078][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 13, column 1 (byte 170) after #input {\n#    tcp {\n#        port => \"3999\"\n#    }   \n#}\n#\n#output {\n#  elasticsearch {\n#    hosts => [ \"localhost:9200\" ]\n#    index => [\"test-%{+YYYY.MM:DD}\"]\n#  }\n#}\n", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:47:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:55:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:17:in `block in compile_sources'", "org/jruby/RubyArray.java:2580:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:14:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:161:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:27:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326:in `block in converge_state'"]}
[2020-02-11T21:30:28,776][INFO ][org.reflections.Reflections] Reflections took 43 ms to scan 1 urls, producing 20 keys and 40 values 
[2020-02-11T21:30:28,961][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch bulk_path=>"/_monitoring/bulk?system_id=logstash&system_api_version=7&interval=1s", hosts=>[http://localhost:9200], sniffing=>false, manage_template=>false, id=>"3a1c66fe0c4c2004551276421f21bb3f18360566c08bee84dc6bc165df2cfa94", document_type=>"%{[@metadata][document_type]}", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_a4e79c4d-f0d3-4b1e-954c-f65aa448af2a", enable_metric=>true, charset=>"UTF-8">, workers=>1, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, ilm_enabled=>"auto", ilm_rollover_alias=>"logstash", ilm_pattern=>"{now/d}-000001", ilm_policy=>"logstash-policy", action=>"index", ssl_certificate_verification=>true, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
[2020-02-11T21:30:29,091][INFO ][logstash.outputs.elasticsearch][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2020-02-11T21:30:29,122][WARN ][logstash.outputs.elasticsearch][.monitoring-logstash] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2020-02-11T21:30:29,133][INFO ][logstash.outputs.elasticsearch][.monitoring-logstash] ES Output version determined {:es_version=>7}
[2020-02-11T21:30:29,140][WARN ][logstash.outputs.elasticsearch][.monitoring-logstash] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2020-02-11T21:30:29,182][INFO ][logstash.outputs.elasticsearch][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[2020-02-11T21:30:29,290][INFO ][logstash.javapipeline    ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x7e1e63f7 run>"}
[2020-02-11T21:30:29,660][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 13, column 1 (byte 170) after #input {\n#    tcp {\n#        port => \"3999\"\n#    }   \n#}\n#\n#output {\n#  elasticsearch {\n#    hosts => [ \"localhost:9200\" ]\n#    index => [\"test-%{+YYYY.MM:DD}\"]\n#  }\n#}\n", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:47:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:55:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:17:in `block in compile_sources'", "org/jruby/RubyArray.java:2580:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:14:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:161:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:27:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326:in `block in converge_state'"]}
[2020-02-11T21:30:29,967][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2020-02-11T21:30:30,211][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-02-11T21:30:31,088][INFO ][logstash.javapipeline    ] Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
[2020-02-11T21:30:31,237][INFO ][logstash.runner          ] Logstash shut down.

This seems to point to a configuration file, however, I haven't changed any config prior to the update.

Thanks in advance for your help.

[Badger]

OK, so your configuration starts with

#input {
#    tcp {
#        port => "3999"
#    }   
#}
#
#output {
#  elasticsearch {
#    hosts => [ "localhost:9200" ]
#    index => ["test-%{+YYYY.MM:DD}"]
#  }
#}

Those are the first twelve lines. Whatever comes next is the problem.

[andre22]

You were right. I found this file with only these lines, all uncommented and moved it away. Nothing else - now logstash is started.
Weird

Thank you very much!

[rugenl (Len Rugen)]

I think some package managers get confused or thing get bundled incorrectly.

If the package updates logstash.yml (and it doesn't match the package contents), sometimes it creates logstash.yml_rpmnew, and leaves logstash.yml alone. Sometimes it creates logstash.yml_rpmsave and replaces logstash.yml.

[andre22]

I think I know what happened. I updated the server yesterday when the SSH connection was interrupted. After this I had to fix the update process, and didn't get an option to keep the existing config files. I'll get them from a backup.
Thanks for your answer!