Xml filter: Unable to get parent and child attributes in one row


(robert) #1

Hi Team
I am new to Kibana, logstash. I am trying push xml (generated as output of nunit) in logstash. I want to have the xml elements attributes, its Parents attributes and child attributes in the same row in Kibana.
However, i can get only the xml elements attribute. Parent and child attributes are not generated in the same row.

Below is the xml file:

<test-suite type="TestFixture" name="bootAPIs" executed="True" result="Failure" success="False" time="12.811" asserts="0">
  <results>
    <test-case name="NunitBETests.bootAPIs.GetSeriescount" executed="True" result="Success" success="True" time="0.944" asserts="0">
      <reason name="NeededforBooting">
        <message />
      </reason>
    </test-case>
    <test-case name="NunitBETests.bootAPIs.GetData" executed="True" result="Success" success="True" time="0.910" asserts="0">
      <reason>
        <message />
      </reason>
    </test-case>
    <test-case name="NunitBETests.bootAPIs.GetToken" executed="True" result="Success" success="True" time="0.462" asserts="0">
      <reason>
        <message />
        e
      </reason>
    </test-case>
  </results>
</test-suite>

Below is Logstash Config File I am using:

input { stdin { } }
filter {
  xml {
   store_xml => false
   source => "message"
   xpath =>
   [
    "//test-case/@name", "testcase",
	"//test-case/@result", "res",
	"//test-case/../../@name", "suitename",
	"//test-case/reason/@name","reasonattr",
	"//test-case/@time", "timetest"
   ]
}
 
date {
    match => [ "date" , "dd-MM-yyyy HH:mm:ss" ]
    timezone => "Europe/Amsterdam"
}

 
}

output {
 
 elasticsearch {
  action => "index"
  index => "xml12"
  hosts => ["localhost:9200"]
  
 
 }
 
 stdout { codec => rubydebug }
}

Result:
I am getting this output:

{
       "message" => "    <test-case name=\"NunitBETests.bootAPIs.GetSeriescount\" executed=\"True\" result=\"Success\" success=\"True\" time=\"0.462\" asserts=\"0\">\r",
      "testcase" => [
        [0] "NunitBETests.bootAPIs.GetSeriescount"
    ],
           "res" => [
        [0] "Success"
    ],

          "host" => "DESKTOP-PC8JBMK"
}

However I am want to capture parent attribute of "testcase" node (that is "suitename") and child attribute of "testcase" xmlnode (that is "reasonattr") in same row.
Something like below is what i want to have:

{
       "message" => "    <test-case name=\"NunitBETests.bootAPIs.GetSeriescount\" executed=\"True\" result=\"Success\" success=\"True\" time=\"0.462\" asserts=\"0\">\r",
      "testcase" => [
        [0] "NunitBETests.bootAPIs.GetSeriescount"
    ],
           "res" => [
        [0] "Success"
    ],
	        "suitename" => [
        [0] "bootAPIs"
    ],
	    "reasonattr" => [
        [0] "NeededforBooting"
    ],
		
    "@timestamp" => 2018-01-09T05:53:07.678Z,
      "@version" => "1",
      "timetest" => [
        [0] "0.462"
    ],
          "host" => "DESKTOP-PC8JBMK"
}

Please let me know what I am missing here.


(robert) #3

Uploading the xml file as png, as i was unable to paste it here: Apologies!!!


(Guy Boertje) #4

I think that because you are not feeding the whole xml file into stdin, the xpath list is failing on the ../../ and not processing the rest. Try moving the "//test-case/../../@name", "suitename", to the bottom.
Aslo try feeding the full xml into stdin.
see "message" => " <test-case


(Guy Boertje) #5

With the full xml message, this worked for me...

filter {
  xml {
   store_xml => false
   source => "message"
   xpath =>
   [
    "/test-suite/@name", "suitename",
    "//test-case[1]/@name", "testcase",
    "//test-case[1]/@result", "res",
    "//test-case[1]/reason/@name","reasonattr",
    "//test-case[1]/@time", "timetest"
   ]
  }
}

(robert) #6

thanks Guy!!!, you rightly pointed out that the xml that I was passing was not a complete "message"... helps a lot.

Now the xml file is processed as a single unit, however am yet not able to get the expected result.

I tried options like
force_array => false, force_content => true
and variations in xpath like:

xpath =>

[
"/test-suite/@name", "suitename",
"//test-case[1]/@name", "testcase",
"//test-case[1]/@result", "res",
"//test-case[1]/reason/@name","reasonattr",
"//test-case[1]/@time", "timetest"
]

xpath =>
[
"/test-suite/@name", "suitename",
"//test-case[]/@name", "testcase",
"//test-case[
]/@result", "res",
"//test-case[]/reason/@name","reasonattr",
"//test-case[
]/@time", "timetest"
]

xpath =>
[
"/test-suite/results/test-case/@name","testcase",
"/test-suite/@name", "suitename"
]

but i got output like below in almost all cases, here the data of all testcases is packed in a single row:

{
"suitename" => [
[0] "bootAPIs"
],
"timetest" => [
[ 0] 1.582,
[ 1] 0.939,
[ 2] 0.91,
[ 3] 0.921,
[ 4] 1.574,
[ 5] 0.908,
[ 6] 0.913,
[ 7] 1.787,
[ 8] 0.903,
[ 9] 0.944,
[10] 0.91,
[11] 0.462
],
"testcase" => [
[ 0] "NunitBETests.bootAPIs.GetSeriesContent",
[ 1] "NunitBETests.bootAPIs.GetSeriesContent",
[ 2] "NunitBETests.bootAPIs.GetToken",
[ 3] "NunitBETests.bootAPIs.GetDomainPermittedSubscriptions",
[ 4] "NunitBETests.bootAPIs.GetEPGMultiChannelProgram",
[ 5] "NunitBETests.bootAPIs.GetItemFromList",
[ 6] "NunitBETests.bootAPIs.GetMenu",
[ 7] "NunitBETests.bootAPIs.GetRecordings",
[ 8] "NunitBETests.bootAPIs.GetSecuredSiteGuid",
[ 9] "NunitBETests.bootAPIs.GetSeriesRecordings",
[10] "NunitBETests.bootAPIs.GetUserData",
[11] "NunitBETests.bootAPIs.RefreshAccessToken"
],
"@version" => "1",
"host" => "DESKTOP-PC8JBMK",
"res" => [
[ 0] "Failure",
[ 1] "Success",
[ 2] "Success",
[ 3] "Success",
[ 4] "Success",
[ 5] "Success",
[ 6] "Success",
[ 7] "Success",
[ 8] "Success",
[ 9] "Success",
[10] "Success",
[11] "Success"
],

What I am wanting is each of the testcase attribute, Suite Name attribute (parent of testcase) to be collected as a single row for EACH testcase present in xml file.
Something like below is what i Want to get:

	  {
		 "suitename" => [
			[0] "bootAPIs"
		],
		  "timetest" => [
			[ 0] 1.582,
			
		],
		  "testcase" => [
			[ 0] "NunitBETests.bootAPIs.GetSeriescContent",
			
		],
		  "@version" => "1",
			  "host" => "DESKTOP-PC8JBMK",
			   "res" => [
			[ 0] "Failure",
		   
		],
	}

	{
		 "suitename" => [
			[1] "bootAPIs"
		],
		  "timetest" => [
			[ 1] 0.939,
			
		],
		  "testcase" => [
			[ 1] ""NunitBETests.bootAPIs.bootAPIs.GetSeriescContent"",
			
		],
		  "@version" => "1",
			  "host" => "DESKTOP-PC8JBMK",
			   "res" => [
			[ 1] "Success",
		   
		],
	}

	{
		 "suitename" => [
			[2] "bootAPIs"
		],
		  "timetest" => [
			[ 2] 0.91,
			
		],
		  "testcase" => [
			[ 2] "NunitBETests.bootAPIs.GetToken"",
			
		],
		  "@version" => "1",
			  "host" => "DESKTOP-PC8JBMK",
			   "res" => [
			[ 2] "Success",
		   
		],
	}

Please let me know, how I could get this done.


(Guy Boertje) #7

Maybe. If you really got to know XPath.

It seems to me that you want separate events for each test-case but with the @name of test-suite carried into each test-case event.

I think this can be done. It would involve two xml filter stages with a split filter in between.
Use this online XPath tester

  1. 1st xml filter, add two fields
    "//test-case/../../@name", "suitename",
    "//test-case", "test-cases"
  2. Split filter, split on test-cases
    field => "test-cases"
    target => "test-case"
    this should create X number of new events all with suitename and test-case with xml content`.
  3. 2nd xml filter, add the other fields.
    "/test-case/@name", "testcase",
    "/test-case/@result", "res",
    "/test-case/reason/@name","reasonattr",
    "/test-case/@time", "timetest"
    Remember to drop any fields you no longer need.
    Now elasticsearch and kibana have a single doc structure of test case to work with.

Good Luck.


Add new elements in XML array
(robert) #8

Thanks a ton Guy!!!

With all your suggestions I could finally come up with what i wanted.

Pasting here the complete details, just in case that it may assit some one else tooo...

Thanks again!

Problem statement:
Below xml is an output generate out of Nunit . I need to collect attributes of all test-case nodes
along with the attributes of its Parent and child nodes in a single event per testcase so that every event generated by Logstash has testcase name, testsuite name(parent node) and Failure message (child node) for all testcases in the xml.

XMl below:

ycaddztg mg kuxxlc jlsb oagfymbn

Logtash filter below

input { stdin { }

}
filter {
xml {

store_xml => true
target => "poc"
source => "message"

}
split {
field => "poc[results][0][test-suite][0][results][0][test-suite]"
}

split {
field => "poc[results][0][test-suite][0][results][0][test-suite][results][0][test-case]"
}

date {
match => [ "date" , "dd-MM-yyyy HH:mm:ss" ]
timezone => "Europe/Amsterdam"
}

mutate {
add_field => { "number22" => "%{poc[results][0][test-suite][0][results][0][test-suite][name]}" }
add_field => { "name22" => "%{poc[results][0][test-suite][0][results][0][test-suite][results][0][test-case][name]}" }
add_field => { "res22" => "%{poc[results][0][test-suite][0][results][0][test-suite][results][0][test-case][result]}" }
add_field => { "reason22" => "%{poc[results][0][test-suite][0][results][0][test-suite][results][0][test-case][failure][0][message][0]}" }

}

if [reason22] == "%{poc[results][0][test-suite][0][results][0][test-suite][results][0][test-case][failure][0][message][0]}" {
mutate { update => { "reason22" => "" } }
}
}
output {
elasticsearch {
action => "index"
index => "xm64"
hosts => ["localhost:9200"]
}
stdout { codec => rubydebug }
}


(robert) #9
    <test-suite type="Assembly" name="C:\vinayDoNotDelete\ComponentTestallFiles\Softwares\OtherCsharpSOlutionsunitBETests\bin\Debugunitbetests.exe" executed="True" result="Failure" success="False" time="18.987" asserts="0">
      <results>
        <test-suite type="Namespace" name="NunitBETests" executed="True" result="Failure" success="False" time="18.978" asserts="0">
          <results>
            <test-suite type="TestFixture" name="bootAPIs" executed="True" result="Failure" success="False" time="12.811" asserts="0">
              <results>
                <test-case name="NunitBETests.bootAPIs.GetMOvieFilter" executed="True" result="Failure" success="False" time="1.582" asserts="0">
                  <failure>
                    ycaddztg
                    <message>
                      <![CDATA[Either there are no linear channels or response is incorrect,
     REQUEST is : {"initObj":{"Locale":{"LocaleLanguage":"","LocaleCountry":"","LocaleDevice":"","LocaleUserState":"Unknown"},"Platform":"ConnectedTV","SiteGuid":"958863","DomainID":543335,"UDID":"616439088037","ApiUser":"tvpapi_185si","ApiPass":"A2d4G6","Token":""},"ChannelID":"3407167","picSize":"302x170","pageSize":0,"pageIndex":0,"orderBy":"None","tagsMetas":[],"cutWith":"AND"}
     RESPONSE is : []]]>
                    </message>
                    <stack-trace>
                      <![CDATA[at NunitBETests.bootAPIs.GetMOvieFilter() in c:\vinayDoNotDelete\ComponentTestallFiles\Softwares\OtherCsharpSOlutionsunitBETests\bootAPIs.cs:line 183
    ]]>
                    </stack-trace>
                  </failure>
                </test-case>

                <test-case name="NunitBETests.bootAPIs.GetSubscriptions" executed="True" result="Success" success="True" time="0.921" asserts="0">
                  <reason>
                    <message />
                  </reason>
                </test-case>
                kvlrw
                <test-case name="NunitBETests.bootAPIs.GetProgram" executed="True" result="Success" success="True" time="1.574" asserts="0">
                  <reason>
                    <message />
                  </reason>
                </test-case>

                <test-case name="NunitBETests.bootAPIs.GetSchedule" executed="True" result="Success" success="True" time="1.787" asserts="0">
                  <reason>
                    dxgarfj
                    <message />
                  </reason>
                </test-case>
                <test-case name="NunitBETests.bootAPIs.GetGuid" executed="True" result="Success" success="True" time="0.903" asserts="0">
                  <reason>
                    <message />
                  </reason>
                </test-case>
                <test-case name="NunitBETests.bootAPIs.GetAllVeggies" executed="True" result="Success" success="True" time="0.944" asserts="0">
                  <reason>
                    <message />
                  </reason>
                </test-case>
                <test-case name="NunitBETests.bootAPIs.GetAllNoNVegies" executed="True" result="Success" success="True" time="0.910" asserts="0">
                  <reason>
                    <message />
                  </reason>
                </test-case>
                <test-case name="NunitBETests.bootAPIs.Refresh" executed="True" result="Success" success="True" time="0.462" asserts="0">
                  <reason>
                    <message />
                  </reason>
                </test-case>
              </results>
            </test-suite>
            <test-suite type="TestFixture" name="epgAPIs" executed="True" result="Failure" success="False" time="3.392" asserts="0">
              <results>
                <test-case name="NunitBETests.epgAPIs.GetMOvieFilter" executed="True" result="Failure" success="False" time="0.912" asserts="0">
                  <failure>
                    <message>
                      <![CDATA[Either there are no series recordings or response incorrect, response is : []]]>
                    </message>
                    kuxxlc
                    <stack-trace>
                      <![CDATA[at NunitBETests.epgAPIs.GetMOvieFilter() in c:\vinayDoNotDelete\ComponentTestallFiles\Softwares\OtherCsharpSOlutionsunitBETests\epgAPIs.cs:line 43
    ]]>
                    </stack-trace>
                  </failure>
                </test-case>
                <test-case name="NunitBETests.epgAPIs.GetProgram" executed="True" result="Success" success="True" time="1.565" asserts="0">
                  <reason>
                    <message />
                  </reason>
                </test-case>
                jlsb
                <test-case name="NunitBETests.epgAPIs.GetData" executed="True" result="Success" success="True" time="0.908" asserts="0">
                  <reason>
                    <message />
                  </reason>
                </test-case>
              </results>
            </test-suite>
            <test-suite type="TestFixture" name="recordingsAPIs" executed="True" result="Success" success="True" time="2.759" asserts="0">
              <results>
                <test-case name="NunitBETests.recordingsAPIs.GetRecordings" executed="True" result="Success" success="True" time="1.810" asserts="0">
                  <reason>
                    <message />
                  </reason>
                </test-case>
                <test-case name="NunitBETests.recordingsAPIs.GetVeggies" executed="True" result="Success" success="True" time="0.944" asserts="0">
                  <reason>
                    <message />
                  </reason>
                </test-case>
                oagfymbn
              </results>
            </test-suite>
          </results>
        </test-suite>
      </results>
    </test-suite>

(system) #10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.