So i'm wrestling with a new problem today... Logstash 7.5.0
recently upgraded my test ELK stack from 7.4.2 to 7.5.0
Immediately hit an issue ingestion data with my setup.
the error i get is
Dec 03 15:14:49 ELK01 logstash[20424]: [2019-12-03T15:14:49,217][DEBUG][logstash.agent ] Converging pipelines state {:actions_count=>1}
Dec 03 15:14:49 ELK01 logstash[20424]: [2019-12-03T15:14:49,224][DEBUG][logstash.agent ] Executing action {:action=>LogStash::PipelineAction::Create/pipeline_id:main}
Dec 03 15:14:49 ELK01 logstash[20424]: [2019-12-03T15:14:49,285][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
Dec 03 15:14:49 ELK01 logstash[20424]: [2019-12-03T15:14:49,290][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
Dec 03 15:14:49 ELK01 logstash[20424]: [2019-12-03T15:14:49,308][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :**message=>"Expected one of [ \\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 1, column 1 (byte 322) after #- pipeline.id: main\n#** path.config: \"/etc/logstash/conf.d/*.conf\"\n# pipeline.workers: 1\n", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2584:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:156:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:26:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326:in `block in converge_state'"]}
Dec 03 15:14:54 ELK01 logstash[20424]: [2019-12-03T15:14:54,317][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
Dec 03 15:14:54 ELK01 logstash[20424]: [2019-12-03T15:14:54,324][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
My pipelines.yml is thus:
- pipeline.id: dataingest
path.config: "/etc/logstash/conf.d/*.conf"
pipeline.workers: 1
And my logstash.yml is thus:
pipeline.batch.size: 125
pipeline.batch.delay: 5
node.name: logstash
path.data: /var/lib/logstash
pipeline.unsafe_shutdown: true
path.config: /etc/logstash/pipelines.yml
config.reload.automatic: true
config.reload.interval: 10s
config.debug: true
http.host: "127.0.0.1"
http.port: 9600-9700
log.level: debug
path.logs: /var/log/logstash
pipeline.separate_logs: true
I've checked with the 7.5.0 guides and there's nothing there comparing to my configs thats amiss... yet the error message=>"Expected one of [ \t\r\n], "#", "input", "filter", "output" at line 1, column 1 (byte 322) after #- pipeline.id: main\n# is basically complaining that there's no config.string: "input { } filter { } output { }" like entry.... but we have these in our logstash configs already... has there been a massive change in how the process works that hasn't yet made it into the logstash guides yet?