Invalid FieldReference: `_Domain_Labels[0]_ULabel`

Elastic Stack Logstash
1

Logstash version - 7.17.8

Currently we are seeing invalid FieldReference errors on our Logstash nodes dealing with _Domain_Labels[0]_ULabel

onf.d/mulesoft/get_cloudhub_app_logs.conf"], :thread=>"#<Thread:0x1475cac0 run>"}
warning: thread "Ruby-0-Thread-71: :1" terminated with exception (report_on_exception is true):
org.logstash.FieldReference$IllegalSyntaxException: Invalid FieldReference: `_Domain_Labels[0]_ULabel`
        at org.logstash.FieldReference$StrictTokenizer.tokenize(FieldReference.java:321)
        at org.logstash.FieldReference.parse(FieldReference.java:222)
        at org.logstash.FieldReference.parseToCache(FieldReference.java:213)
        at org.logstash.FieldReference.from(FieldReference.java:136)
        at org.logstash.FieldReference.from(FieldReference.java:122)
        at org.logstash.ConvertedMap.convertKey(ConvertedMap.java:121)
        at org.logstash.ConvertedMap.access$000(ConvertedMap.java:43)
        at org.logstash.ConvertedMap$1.visit(ConvertedMap.java:54)
        at org.logstash.ConvertedMap$1.visit(ConvertedMap.java:48)
        at org.jruby.RubyHash.visitLimited(RubyHash.java:698)
        at org.jruby.RubyHash.visitAll(RubyHash.java:683)
        at org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:88)
        at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_initialize(JrubyEventExtLibrary.java:97)
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_gelf_minus_3_dot_3_dot_2.lib.logstash.inputs.gelf.RUBY$method$parse$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:253)
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_gelf_minus_3_dot_3_dot_2.lib.logstash.inputs.gelf.RUBY$method$new_event$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:226)
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_gelf_minus_3_dot_3_dot_2.lib.logstash.inputs.gelf.RUBY$method$udp_listener$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:206)
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_gelf_minus_3_dot_3_dot_2.lib.logstash.inputs.gelf.RUBY$method$udp_listener$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:180)
        at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)
        at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)
        at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_gelf_minus_3_dot_3_dot_2.lib.logstash.inputs.gelf.RUBY$block$run$2(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:79)
        at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)
        at org.jruby.runtime.Block.call(Block.java:139)
        at org.jruby.RubyProc.call(RubyProc.java:318)
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)
        at java.base/java.lang.Thread.run(Thread.java:829)
[2023-06-27T14:12:43,361][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>org.logstash.FieldReference$IllegalSyntaxException: Invalid FieldReference: `_Domain_Labels[0]_ULabel`, :backtrace=>["org.logstash.FieldReference$StrictTokenizer.tokenize(FieldReference.java:321)", "org.logstash.FieldReference.parse(FieldReference.java:222)", "org.logstash.FieldReference.parseToCache(FieldReference.java:213)", "org.logstash.FieldReference.from(FieldReference.java:136)", "org.logstash.FieldReference.from(FieldReference.java:122)", "org.logstash.ConvertedMap.convertKey(ConvertedMap.java:121)", "org.logstash.ConvertedMap.access$000(ConvertedMap.java:43)", "org.logstash.ConvertedMap$1.visit(ConvertedMap.java:54)", "org.logstash.ConvertedMap$1.visit(ConvertedMap.java:48)", "org.jruby.RubyHash.visitLimited(RubyHash.java:698)", "org.jruby.RubyHash.visitAll(RubyHash.java:683)", "org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:88)", "org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_initialize(JrubyEventExtLibrary.java:97)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_gelf_minus_3_dot_3_dot_2.lib.logstash.inputs.gelf.RUBY$method$parse$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:253)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_gelf_minus_3_dot_3_dot_2.lib.logstash.inputs.gelf.RUBY$method$new_event$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:226)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_gelf_minus_3_dot_3_dot_2.lib.logstash.inputs.gelf.RUBY$method$udp_listener$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:206)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_gelf_minus_3_dot_3_dot_2.lib.logstash.inputs.gelf.RUBY$method$udp_listener$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:180)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_gelf_minus_3_dot_3_dot_2.lib.logstash.inputs.gelf.RUBY$block$run$2(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.2/lib/logstash/inputs/gelf.rb:79)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]}
[2023-06-27T14:12:43,476][INFO ][logstash.javapipeline    ][beats] Pipeline Java execution initialization time {"seconds"=>2.26}
[2023-06-27T14:12:43,505][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]

I have tried adding in our Logstash config the mutator below but still get errors after restarting logstash:

mutate {
        rename => { "_Domain_Labels[0]_ULabel " => "%{[Domain_Labels[0]_ULabel]}"}
    }

Is there a way to mutate _Domain_Labels[0]_ULabel to fix the invalid FieldReference error?

2

By the time a field with that name is created it may already be too late to fix it. How is it getting created?

3

Its getting created via another project we have that turns it into _Domain_Labels_0_ULabel. So I think your right its probably too late to fix using mutate, and we probably have to find the source and change config there.

4

personally curious. Do you mind providing a sample doc?

5

@Badger Is there a way to ignore invalid fieldreference errors?

6

I think that only if you do not try to mutate the field, if you try to mutate you will get this error.

I don't think you can fix this in Logstash, you need to fix this before sending the data to Logstash.