XPath issue with different XML files

AurelienG · June 11, 2018, 2:53pm

Hello,
Here's my problem: I have several XML files, which don't have the same structure, and I need to get one of the fields, which is not even always there... So here's what I've tried:

#Check if there's a jdk field
if "<jdk>" in [config] {
	xml {
		source => "config"
		store_xml => "false"
		xpath => ["*/jdk/text()", "jdk-version"]
	}
}

else if "JDKAxis" in [config] {
	xml {
		source => "config"
		store_xml => "false"
		xpath => ["//hudson.matrix.JDKAxis/values/string/text()", "jdk-version"]
	}
}
	
# If there isn't, add the field with NULL value
else {
	mutate {
		add_field => ["jdk-version", "NULL"]
	}
}

The XPath syntax is right, I have already tried on XPath testers. I have also tried without /text(), it didn't change anything.
The field is created in my index pattern but is empty for every created document.

Am I doing something wrong?

Badger · June 11, 2018, 3:26pm

What does the raw xml look like?

AurelienG · June 12, 2018, 7:15am

Well one of them looks like that, with more fields but I don't think it's useful to put everything, it doesn't change the XPath.

<exml version='1.0' encoding='UTF-8'e>
<maven2-moduleset plugin="maven-plugin@3.0">
  <actions/>
  <description></description>
  <jdk>(System)</jdk>
</maven2-moduleset>

Here's another one:

<exml version='1.0' encoding='UTF-8'e>
<matrix-project plugin="matrix-project@1.12">
  <actions/>
  <description></description>
  <axes>
    <hudson.matrix.TextAxis>
      <name>MTrack</name>
      <values>
        <string>GEN</string>
        <string>MINI-MEM</string>
      </values>
    </hudson.matrix.TextAxis>
    <hudson.matrix.JDKAxis>
      <name>jdk</name>
      <values>
        <string>jdk-1.7</string>
      </values>
    </hudson.matrix.JDKAxis>
  </axes>
</matrix-project>

And the other structures don't have the information I'm interested in.

AurelienG · June 13, 2018, 2:23pm

I still haven't found anything to solve my problem, does anyone have an idea?

Badger · June 13, 2018, 3:16pm

No, it does not change the xpath, but the problem is not with the xpath. It is that exml line at the top of your file. I would strip that off.

mutate { gsub => [ "message", "^<exml[^>]+>", "" ] }

Note that using xpath always gets you an array.

AurelienG · June 14, 2018, 9:52am

It actually never gave me an error about bad XML format, but you're right, it might be wrong.
So I tried this and now I have a huge error in the log, and I can't figure out what it is about.

[ERROR][logstash.pipeline        ] xedModeIRMethod.java:95)", 
"org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:77)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", 
"org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", 
"org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", 
"org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:186)", 
"org.jruby.runtime.BlockBody.yield(BlockBody.java:116)", 
"org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyHash$12.visit(RubyHash.java:1362)", 
"org.jruby.RubyHash$12.visit(RubyHash.java:1359)", 
"org.jruby.RubyHash.visitLimited(RubyHash.java:662)", 
"org.jruby.RubyHash.visitAll(RubyHash.java:647)", 
"org.jruby.RubyHash.iteratorVisitAll(RubyHash.java:1319)", 
"org.jruby.RubyHash.each_pairCommon(RubyHash.java:1354)", 
"org.jruby.RubyHash.each(RubyHash.java:1343)", 
"org.jruby.RubyHash$INVOKER$i$0$0$each.call(RubyHash$INVOKER$i$0$0$each.gen)", 
"org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498)", 
"org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:77)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", 
"org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", 
"org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:171)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:177)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.invokeOther4:filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$do_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145)", 
"org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)", 
"org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:171)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:177)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.invokeOther4:do_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$block$multi_filter$1(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164)", 
"org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)", 
"org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", 
"org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", 
"org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)", 
"org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:139)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:145)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.invokeOther7:each(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$multi_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161)", 
"org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filter_delegator.invokeOther10:multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:47)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filter_delegator.RUBY$method$multi_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:47)", 
"org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", 
"org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", 
"org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:186)", 
"org.jruby.runtime.BlockBody.yield(BlockBody.java:116)", 
"org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", 
"org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)", 
"org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498)", 
"org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)", 
"org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", 
"org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", 
"org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)",

AurelienG · June 14, 2018, 9:53am

The error didn't fit in one message so here's the end:

"org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", 
"org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", 
"org.jruby.RubyProc.call19(RubyProc.java:273)", 
"org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:204)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", 
"org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", 
"org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", 
"org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", 
"org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", 
"org.jruby.internal.runtime.methods.ProcMethod.call(ProcMethod.java:63)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:204)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", 
"org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", 
"org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:89)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:214)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:200)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208)", 
"org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:358)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:195)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:323)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", 
"org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", 
"org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", 
"org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", 
"org.jruby.RubyProc.call(RubyProc.java:246)", 
"org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)", 
"java.lang.Thread.run(Thread.java:748)"], 
:thread=>"#<Thread:0x1ec41443@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 sleep>"}

AurelienG · June 19, 2018, 2:39pm

I tried again today and got an error I've already had before: "exception"=>"ASCII-8BIT"
That was because of interrogation marks in my XML file I think. Last time I solved it with a gsub but this time it didn't seem to work so I force encoded to UTF-8 in a ruby filter.
Not only did it solve the encoding problem, but it also made my XPath work. So this problem is solved but I don't feel like I have really found the solution, it kinda looks like magic to me!

Anyway, thank you for your time

system · July 17, 2018, 2:39pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.