XPath issue with different XML files

AurelienG · June 11, 2018, 2:53pm

Hello,
Here's my problem: I have several XML files, which don't have the same structure, and I need to get one of the fields, which is not even always there... So here's what I've tried:

#Check if there's a jdk field
if "<jdk>" in [config] {
	xml {
		source => "config"
		store_xml => "false"
		xpath => ["*/jdk/text()", "jdk-version"]
	}
}

else if "JDKAxis" in [config] {
	xml {
		source => "config"
		store_xml => "false"
		xpath => ["//hudson.matrix.JDKAxis/values/string/text()", "jdk-version"]
	}
}
	
# If there isn't, add the field with NULL value
else {
	mutate {
		add_field => ["jdk-version", "NULL"]
	}
}

The XPath syntax is right, I have already tried on XPath testers. I have also tried without /text(), it didn't change anything.
The field is created in my index pattern but is empty for every created document.

Am I doing something wrong?

Badger · June 11, 2018, 3:26pm

What does the raw xml look like?

AurelienG · June 12, 2018, 7:15am

Well one of them looks like that, with more fields but I don't think it's useful to put everything, it doesn't change the XPath.

<exml version='1.0' encoding='UTF-8'e>
<maven2-moduleset plugin="maven-plugin@3.0">
  <actions/>
  <description></description>
  <jdk>(System)</jdk>
</maven2-moduleset>

Here's another one:

<exml version='1.0' encoding='UTF-8'e>
<matrix-project plugin="matrix-project@1.12">
  <actions/>
  <description></description>
  <axes>
    <hudson.matrix.TextAxis>
      <name>MTrack</name>
      <values>
        <string>GEN</string>
        <string>MINI-MEM</string>
      </values>
    </hudson.matrix.TextAxis>
    <hudson.matrix.JDKAxis>
      <name>jdk</name>
      <values>
        <string>jdk-1.7</string>
      </values>
    </hudson.matrix.JDKAxis>
  </axes>
</matrix-project>

And the other structures don't have the information I'm interested in.

AurelienG · June 13, 2018, 2:23pm

I still haven't found anything to solve my problem, does anyone have an idea?

Badger · June 13, 2018, 3:16pm

No, it does not change the xpath, but the problem is not with the xpath. It is that exml line at the top of your file. I would strip that off.

mutate { gsub => [ "message", "^<exml[^>]+>", "" ] }

Note that using xpath always gets you an array.

AurelienG · June 14, 2018, 9:52am

It actually never gave me an error about bad XML format, but you're right, it might be wrong.
So I tried this and now I have a huge error in the log, and I can't figure out what it is about.

[ERROR][logstash.pipeline        ] xedModeIRMethod.java:95)", 
"org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:77)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", 
"org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", 
"org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", 
"org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:186)", 
"org.jruby.runtime.BlockBody.yield(BlockBody.java:116)", 
"org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyHash$12.visit(RubyHash.java:1362)", 
"org.jruby.RubyHash$12.visit(RubyHash.java:1359)", 
"org.jruby.RubyHash.visitLimited(RubyHash.java:662)", 
"org.jruby.RubyHash.visitAll(RubyHash.java:647)", 
"org.jruby.RubyHash.iteratorVisitAll(RubyHash.java:1319)", 
"org.jruby.RubyHash.each_pairCommon(RubyHash.java:1354)", 
"org.jruby.RubyHash.each(RubyHash.java:1343)", 
"org.jruby.RubyHash$INVOKER$i$0$0$each.call(RubyHash$INVOKER$i$0$0$each.gen)", 
"org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498)", 
"org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:77)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", 
"org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", 
"org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:171)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:177)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.invokeOther4:filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$do_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145)", 
"org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)", 
"org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:171)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:177)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.invokeOther4:do_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$block$multi_filter$1(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164)", 
"org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)", 
"org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", 
"org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", 
"org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)", 
"org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:139)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:145)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.invokeOther7:each(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$multi_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161)", 
"org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filter_delegator.invokeOther10:multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:47)", 
"usr.share.logstash.logstash_minus_core.lib.logstash.filter_delegator.RUBY$method$multi_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:47)", 
"org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:161)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", 
"org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", 
"org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:186)", 
"org.jruby.runtime.BlockBody.yield(BlockBody.java:116)", 
"org.jruby.runtime.Block.yield(Block.java:165)", "org.jruby.RubyArray.each(RubyArray.java:1734)", 
"org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)", 
"org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498)", 
"org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)", 
"org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", 
"org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", 
"org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)",

AurelienG · June 14, 2018, 9:53am

The error didn't fit in one message so here's the end:

"org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", 
"org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", 
"org.jruby.RubyProc.call19(RubyProc.java:273)", 
"org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:204)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", 
"org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", 
"org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", 
"org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", 
"org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", 
"org.jruby.internal.runtime.methods.ProcMethod.call(ProcMethod.java:63)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:204)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", 
"org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:83)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:179)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:165)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)", 
"org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:89)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:214)", 
"org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:200)", 
"org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208)", 
"org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:358)", 
"org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:195)", 
"org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:323)", 
"org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", 
"org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)", 
"org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)", 
"org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)", 
"org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", 
"org.jruby.RubyProc.call(RubyProc.java:246)", 
"org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)", 
"java.lang.Thread.run(Thread.java:748)"], 
:thread=>"#<Thread:0x1ec41443@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 sleep>"}

AurelienG · June 19, 2018, 2:39pm

I tried again today and got an error I've already had before: "exception"=>"ASCII-8BIT"
That was because of interrogation marks in my XML file I think. Last time I solved it with a gsub but this time it didn't seem to work so I force encoded to UTF-8 in a ruby filter.
Not only did it solve the encoding problem, but it also made my XPath work. So this problem is solved but I don't feel like I have really found the solution, it kinda looks like magic to me!

Anyway, thank you for your time

system · July 17, 2018, 2:39pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Mutiple xml file with same fields (How to handle?) any suggestion Logstash	4	861	July 6, 2017
Created nested fields from Xpath & check for existing documents Logstash	3	491	February 11, 2020
Parse XML file question Logstash	9	737	April 24, 2018
Help Logstash XML Parsing to Xpath Logstash	23	1392	September 11, 2020
Xpath fails to extract Logstash	5	207	January 2, 2023

XPath issue with different XML files

Related topics