Config has duplicate Ids

[bmh.01]

Has anyone ran into this before? The only thing I can find is this bug report that seems to be untouched from the alpha days.

https://github.com/elastic/logstash/issues/7403

Migrating my ELK stack from 5 to 6 and this is the error I'm running into, I've nailed it down to two configuration files which have filter blocks in them and it's the following statement in two of the config files that seems to be causing it.

if !("_grokparsefailure" in [tags]) {
    do some stuff
}

The funny thing is I use that statement in several other files and logstash doesn't complain about those! All I need to do is delete one of the 'conflicting' files and it will start up successfully, likewise removing the if statement from one of the files will also allow it to start.

The two files refer to two separate document types as well so this one has completely stumped me at the minute.

[magnusbaeck (Magnus Bäck)]

if !("_grokparsefailure" in [tags]) {

This should work, but the canonical way of expressing it would be:

if "_grokparsefailure" not in [tags] {

[bmh.01]

Ok I didn't realise that i've changed the config to match that but unfortunately it's still referencing the same error which is strange now considering that exact syntax is no longer present. This is the full error below, it makes me wonder where it is getting it from now.

[2017-11-16T16:55:44,936][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::OrgLogstashConfigIr::InvalidIRException", :message=>"Config has duplicate Ids: \nID: 290f61bfbcab108f22b4bb4135da94a4ff94d17a28c0ea88a3f4d63695fed4ef [if !(('_grokparsefailure'.include?event.getField('[tags]')))],[if !(('_grokparsefailure'.include?event.getField('[tags]')))]", :backtrace=>["org.logstash.config.ir.graph.Graph.validate(org/logstash/config/ir/graph/Graph.java:292)", "org.logstash.config.ir.PipelineIR.(org/logstash/config/ir/PipelineIR.java:52)", "java.lang.reflect.Constructor.newInstance(java/lang/reflect/Constructor.java:423)", "org.jruby.javasupport.JavaConstructor.newInstanceDirect(org/jruby/javasupport/JavaConstructor.java:246)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:1022)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "RUBY.compile_sources(/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:30)", "RUBY.compile_lir(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:107)", "RUBY.initialize(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49)", "RUBY.initialize(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:215)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:1022)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "RUBY.execute(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:35)", "RUBY.block in converge_state(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:335)", "RUBY.with_pipelines(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141)", "RUBY.block in converge_state(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:332)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1734)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)", "RUBY.converge_state(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:319)", "RUBY.block in converge_state_and_update(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166)", "RUBY.with_pipelines(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141)", "RUBY.converge_state_and_update(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164)", "RUBY.execute(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90)", "RUBY.block in execute(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:362)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)", "org.jruby.RubyProc.call19(org/jruby/RubyProc.java:273)", "org.jruby.RubyProc$INVOKER$i$0$0$call19.call(org/jruby/RubyProc$INVOKER$i$0$0$call19.gen)", "RUBY.block in initialize(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:246)", "java.lang.Thread.run(java/lang/Thread.java:748)"]}
[2017-11-16T16:55:44,938][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle Java::OrgLogstashConfigIr::InvalidIRException for LogStash::PipelineAction::Create/pipeline_id:main", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/converge_result.rb:27:in create'", "/usr/share/logstash/logstash-core/lib/logstash/converge_result.rb:67:inadd'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:347:in block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:332:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:319:in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:362:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}