Reloading pipeline fails with "Setting \"modules.cli\" hasn't been registered"

I have a logstash pipeline pointed at this configuration

input {}
input { generator { count => 1 lines => [ '' ] } }
filter {
}
output { stdout { codec => rubydebug { metadata => false } } }

I then updated my pipelines.yml to contain

- pipeline.id: main
  path.config: "/bin/bash"

to check the error message I would get. I then flipped the pipelines.yml back to /home/user/test.conf and logstash just loops forever saying

[ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>ArgumentError, :message=>"Setting "modules.cli" hasn't been registered", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:36:in get_setting'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/base.rb:60:inmodules_cli_setting'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/base.rb:64:in modules_cli'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/base.rb:68:inmodules_cli?'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/multi_local.rb:33:in match?'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:48:inblock in fetch'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:92:in block in sources'", "org/jruby/ext/set/RubySet.java:601:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:91:in block in sources'", "org/jruby/ext/thread/Mutex.java:165:insynchronize'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:89:in sources'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:47:infetch'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:148:in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:113:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:18:in interval'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:102:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:367:in block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/task.rb:24:inblock in initialize'"]}

Similar to this (fixed) issue.