XPath issue with different XML files

Hello,
Here's my problem: I have several XML files, which don't have the same structure, and I need to get one of the fields, which is not even always there... So here's what I've tried:

#Check if there's a jdk field
if "<jdk>" in [config] {
	xml {
		source => "config"
		store_xml => "false"
		xpath => ["*/jdk/text()", "jdk-version"]
	}
}

else if "JDKAxis" in [config] {
	xml {
		source => "config"
		store_xml => "false"
		xpath => ["//hudson.matrix.JDKAxis/values/string/text()", "jdk-version"]
	}
}
	
# If there isn't, add the field with NULL value
else {
	mutate {
		add_field => ["jdk-version", "NULL"]
	}
}

The XPath syntax is right, I have already tried on XPath testers. I have also tried without /text(), it didn't change anything.
The field is created in my index pattern but is empty for every created document.

Am I doing something wrong?

What does the raw xml look like?

Well one of them looks like that, with more fields but I don't think it's useful to put everything, it doesn't change the XPath.

<exml version='1.0' encoding='UTF-8'e>
<maven2-moduleset plugin="maven-plugin@3.0">
  <actions/>
  <description></description>
  <jdk>(System)</jdk>
</maven2-moduleset>

Here's another one:

<exml version='1.0' encoding='UTF-8'e>
<matrix-project plugin="matrix-project@1.12">
  <actions/>
  <description></description>
  <axes>
    <hudson.matrix.TextAxis>
      <name>MTrack</name>
      <values>
        <string>GEN</string>
        <string>MINI-MEM</string>
      </values>
    </hudson.matrix.TextAxis>
    <hudson.matrix.JDKAxis>
      <name>jdk</name>
      <values>
        <string>jdk-1.7</string>
      </values>
    </hudson.matrix.JDKAxis>
  </axes>
</matrix-project>

And the other structures don't have the information I'm interested in.

I still haven't found anything to solve my problem, does anyone have an idea?

No, it does not change the xpath, but the problem is not with the xpath. It is that exml line at the top of your file. I would strip that off.

mutate { gsub => [ "message", "^<exml[^>]+>", "" ] }

Note that using xpath always gets you an array.

It actually never gave me an error about bad XML format, but you're right, it might be wrong.
So I tried this and now I have a huge error in the log, and I can't figure out what it is about.

[ERROR][logstash.pipeline        ] xedModeIRMethod.java:95)", 
"org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:77)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", 
"org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", 
"org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", 
"org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:186)", 
"org.jruby.runtime.BlockBody.yield(BlockBody.java:116)", 
"org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyHash$12.visit(RubyHash.java:1362)", 
"org.jruby.RubyHash$12.visit(RubyHash.java:1359)", 
"org.jruby.RubyHash.visitLimited(RubyHash.java:662)", 
"org.jruby.RubyHash.visitAll(RubyHash.java:647)", 
"org.jruby.RubyHash.iteratorVisitAll(RubyHash.java:1319)", 
"org.jruby.RubyHash.each_pairCommon(RubyHash.java:1354)", 
"org.jruby.RubyHash.each(RubyHash.java:1343)", 
"org.jruby.RubyHash$INVOKER$i$0$0$each.call(RubyHash$INVOKER$i$0$0$each.gen)", 
"org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498)", 
"org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:77)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", 
"org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", 
"org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:171)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:177)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.invokeOther4:filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$do_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145)", 
"org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)", 
"org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:171)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:177)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.invokeOther4:do_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$block$multi_filter$1(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164)", 
"org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)", 
"org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", 
"org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", 
"org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)", 
"org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:139)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:145)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.invokeOther7:each(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$multi_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161)", 
"org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filter_delegator.invokeOther10:multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:47)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filter_delegator.RUBY$method$multi_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:47)", 
"org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", 
"org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", 
"org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:186)", 
"org.jruby.runtime.BlockBody.yield(BlockBody.java:116)", 
"org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", 
"org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)", 
"org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498)", 
"org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)", 
"org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", 
"org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", 
"org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", 

The error didn't fit in one message so here's the end:

"org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", 
"org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", 
"org.jruby.RubyProc.call19(RubyProc.java:273)", 
"org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:204)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", 
"org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", 
"org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", 
"org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", 
"org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", 
"org.jruby.internal.runtime.methods.ProcMethod.call(ProcMethod.java:63)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:204)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", 
"org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", 
"org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:89)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:214)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:200)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208)", 
"org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:358)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:195)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:323)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", 
"org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", 
"org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", 
"org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", 
"org.jruby.RubyProc.call(RubyProc.java:246)", 
"org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)", 
"java.lang.Thread.run(Thread.java:748)"], 
:thread=>"#<Thread:0x1ec41443@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 sleep>"}

I tried again today and got an error I've already had before: "exception"=>"ASCII-8BIT"
That was because of interrogation marks in my XML file I think. Last time I solved it with a gsub but this time it didn't seem to work so I force encoded to UTF-8 in a ruby filter.
Not only did it solve the encoding problem, but it also made my XPath work. So this problem is solved but I don't feel like I have really found the solution, it kinda looks like magic to me! :smile:

Anyway, thank you for your time :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.