Logstash configuration is not giving value to add_field

satendr · July 17, 2018, 9:10am

I am trying to Parse XMl file , and adding extra field for test status . file is getting parsed but for New field , instead of getting value , its displaying filter itself. please suggest.. logstash.config file:

input {
  file {
    path => "C:/win10Automation/testng-results.xml"
	start_position => "beginning"
	type => "xml"
	codec => multiline {
                  pattern => "^<\?xmldata .*\>"
				  negate => true
				  what => "previous"
				  
				 
           }
    }
  
  #beats {
	#	port => "5044"
	#}
}
 
filter {
  xml {
  
   remove_namespaces => true
   source => "message"
   store_xml => false
  
  xpath =>
   [
   "/testng-results/suite/test/class/test-method", "test-method",
   "/testng-results/suite/test/class/test-method[3]/@status", "test-status"
   #"/testng-results/suite/test/class/test-method[(@status='PASS' or @status='FAIL' or @status='skipped')  and contains(@signature ,"test()") ]","test-result"
   ]
}

		mutate {
			
            add_field => {"test-method" => "%{test-method]}"}
			add_field => {"test-status" => "%{[test-status]}"}
			
			add_field => {"test-result" => "%{test-result}"}
			
				}
}
 
output {
  elasticsearch {
	codec => json
	hosts => "localhost:9200"
	index => "test10"
  }
 
 stdout {
    codec => rubydebug
  }
 
}

xml file :

<testng-results skipped="0" failed="0" total="10" passed="10">
  <suite name="TestSuite" duration-ms="498075" started-at="2018-06-13T07:00:21Z" finished-at="2018-06-13T07:08:39Z">
	<groups>
	</groups>
	<test name="UIHTML5TC10SettingSupportUI" duration-ms="20867" started-at="2018-06-13T07:00:21Z" finished-at="2018-06-13T07:00:42Z">
	  <class name="com.mtfilemedia.ifive.testscripts.UIHTML5TC10SettingSupportUI">
		<test-method status="PASS" signature="testSetup()[pri:0, instance:com.mtfilemedia.ifive.testscripts.UIHTML5TC10SettingSupportUI@7e0e6aa2]" name="testSetup" is-config="true" duration-ms="14" started-at="2018-06-13T12:30:21Z" finished-at="2018-06-13T12:30:21Z">
		  <reporter-output>
		  </reporter-output>
		</test-method> <!-- testSetup -->
		<test-method status="PASS" signature="beforeMethod()[pri:0, instance:com.mtfilemedia.ifive.testscripts.UIHTML5TC10SettingSupportUI@7e0e6aa2]" name="beforeMethod" is-config="true" duration-ms="24" started-at="2018-06-13T12:30:21Z" finished-at="2018-06-13T12:30:21Z">
		  <reporter-output>
		  </reporter-output>
		</test-method> <!-- beforeMethod -->
		<test-method status="PASS" signature="test()[pri:0, instance:com.mtfilemedia.ifive.testscripts.UIHTML5TC10SettingSupportUI@7e0e6aa2]" name="test" duration-ms="19520" started-at="2018-06-13T12:30:21Z" finished-at="2018-06-13T12:30:40Z">
		</test-method> <!-- test -->
		<test-method status="PASS" signature="afterMethod()[pri:0, instance:com.mtfilemedia.ifive.testscripts.UIHTML5TC10SettingSupportUI@7e0e6aa2]" name="afterMethod" is-config="true" duration-ms="172" started-at="2018-06-13T12:30:41Z" finished-at="2018-06-13T12:30:42Z">
		  <reporter-output>
		  </reporter-output>
		</test-method> <!-- afterMethod -->
		<test-method status="PASS" signature="testTearDown()[pri:0, instance:com.mtfilemedia.ifive.testscripts.UIHTML5TC10SettingSupportUI@7e0e6aa2]" name="testTearDown" is-config="true" duration-ms="2" started-at="2018-06-13T12:30:42Z" finished-at="2018-06-13T12:30:42Z">
		  <reporter-output>
		  </reporter-output>
		</test-method> <!-- testTearDown -->
	  </class> <!-- com.mtfilemedia.ifive.testscripts.UIHTML5TC10SettingSupportUI -->
	</test>

in Kibana we get result test-status = %{[test-status]}, instead of value Pass,Fail, Please help

cjcenizal · July 17, 2018, 8:12pm

Hi there, it looks like there's a syntax error. There's a closing bracket ] but no opening bracket in "%{test-method]}".

If this doesn't solve the problem, keep in mind that the sprintf format won't expand the reference if the field doesn't exist. This would also explain the output you're seeing. If the problem persists after fixing the syntax error, could you verify that the test-status field is making it through your pipeline to your mutate filter?

As a sidenote, why are you using add_field to only reference itself? It seems like you're adding the field test-method with the value of whatever is already in that field.

Also, you can collapse your add_field directive a bit:

mutate {
  add_field => {
    "a" => "b"
    "c" => "d"
  }
}

Hope this helps,
CJ

satendr · July 18, 2018, 5:54am

I have removed add_field method from config file : filter {
xml {
source => "testng-results"
store_xml => "false"

xpath =>
[
"/testng-results/suite/test/class/test-method", "testmethod",
"/testng-results/suite/test/class/test-method[3]/@status", "teststatus"
#"/testng-results/suite/test/class/test-method[(@status='PASS' or @status='FAIL' or @status='skipped') and contains(@signature ,"test()") ]","test-result"
]
}}
I am getting JSON parsed -
\r\n \r\n <suite name="TestSuite" duration-ms="959089" started-at="2018-06-15T05:27:09Z" finished-at="2018-06-15T05:43:09Z">\r\n \r\n \r\n <test name="UIHTML5TC13SettingDeviceOptionsZoom" duration-ms="178857" started-at="2018-06-15T05:27:09Z" finished-at="2018-06-15T05:30:08Z">\r\n <class name="com.slingmedia.safe.testscripts.UIHTML5TC13SettingDeviceOptionsZoom">\r\n <test-method status="PASS" signature="testSetup()[pri:0, instance:com.slingmedia.safe.testscripts.UIHTML5TC13SettingDeviceOptionsZoom@7e0e6aa2]" name="testSetup" is-config="true" duration-ms="27" started-at="2018-06-15T10:57:09Z" finished-at="2018-06-15T10:57:09Z">\r\n \r\n \r\n \r\n <test-method status="PASS" signature="beforeMethod()[pri:0, instance:com.slingmedia.safe.testscripts.UIHTML5TC13SettingDeviceOptionsZoom@7e0e6aa2]" name="beforeMethod" is-config="true" duration-ms="85" started-at="2018-06-15T10:57:10Z" finished-at="2018-06-15T10:57:10Z">\r\n \r\n \r\n \r\n <test-method status="PASS" signature="test()[pri:0, instance:com.slingmedia.safe.testscripts.UIHTML5TC13SettingDeviceOptionsZoom@7e0e6aa2]" name="test" duration-ms="107320" started-at="2018-06-15T10:57:10Z" finished-at="2018-06-15T10:58:57Z">\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r",
"path": "C:/win10Automation/testng-results.xml",
"type": "xml"
}
I want to extract test_status = PASS/FAILED/SKIPPED.
Please help me how to get value

yaauie · July 18, 2018, 5:46pm

The File Input plugin is meant for reading files that are actively being appended to by some outside process (typically one event per line, but as you can see the multiline codec lets it understand groups of lines as a single event), and while it can be made to read whole files that aren't being appended to, it has a couple of quirks.

First, it keeps track of where it left off in each file, and once it has processed a line of input, it won't re-process the same line again. This is what enables it to process append-only log files and emit new events as they arrive, and to not re-emit the same events after a restart. You can tell it to store its "sincedb" on your machine's null device with a sincedb_path directive, which will ensure that it doesn't remember where it left off on files between restarts:

  sincedb_path => "/dev/null"

Next, the File input plugin isn't ever really "done" reading a file that it finds. It reads what is available, and waits for new input to be appended to the end. And since your multiline codec is telling it not to start a new event until it finds a certain pattern, it buffers what is available and then sits there waiting. Because it never reaches a pattern that starts a new event, it doesn't end up emitting your event until you tell the pipeline to shutdown.

The best we can do is tell the multiline codec to flush periodically with an auto_flush_interval directive (setting a value too low can cause a slow input to flush the codec mid-event, so I picked 10 seconds):

  auto_flush_interval => 10

Demonstration of what that means for you in my next post...

yaauie · July 18, 2018, 5:46pm

In the process of attempting to replicate, I saved a file with the entire contents of your "xml file" -- it wasn't quite legal syntax, so I appended the appropriate closing tags and a trailing newline:

  </suite>
</testng-results>

I also noticed that your multiline codec is looking for a line that begins with <?xmldata ... but I didn't see that pattern anywhere in your input. I modified the input to look for a line that begins with <testng-results\b, and added the auto_flush_interval and sincedb_path directives:

input {
  file {
    # path => "C:/win10Automation/testng-results.xml"
    path => "${PWD}/input.xml"
    start_position => "beginning"
    type => "xml"
    sincedb_path => "/dev/null"
    codec => multiline {
      pattern => "^<testng-results\b"
      negate => true
      what => "previous"
      auto_flush_interval => 10 # if we haven't received any more lines in 10s, flush
    }
  }
}

Et viola – It works. We see the test-status field contains PASS, extracted from the given x-path:

{
        "message" => "<testng-results skipped=\"0\" failed=\"0\" total=\"10\" passed=\"10\">\n  <suite name=\"TestSuite\" duration-ms=\"498075\" started-at=\"2018-06-13T07:00:21Z\" finished-at=\"2018-06-13T07:08:39Z\">\n\t<groups>\n\t</groups>\n\t<test name=\"UIHTML5TC10SettingSupportUI\" duration-ms=\"20867\" started-at=\"2018-06-13T07:00:21Z\" finished-at=\"2018-06-13T07:00:42Z\">\n\t  <class name=\"com.mtfilemedia.ifive.testscripts.UIHTML5TC10SettingSupportUI\">\n\t\t<test-method status=\"PASS\" signature=\"testSetup()[pri:0, instance:com.mtfilemedia.ifive.testscripts.UIHTML5TC10SettingSupportUI@7e0e6aa2]\" name=\"testSetup\" is-config=\"true\" duration-ms=\"14\" started-at=\"2018-06-13T12:30:21Z\" finished-at=\"2018-06-13T12:30:21Z\">\n\t\t  <reporter-output>\n\t\t  </reporter-output>\n\t\t</test-method> <!-- testSetup -->\n\t\t<test-method status=\"PASS\" signature=\"beforeMethod()[pri:0, instance:com.mtfilemedia.ifive.testscripts.UIHTML5TC10SettingSupportUI@7e0e6aa2]\" name=\"beforeMethod\" is-config=\"true\" duration-ms=\"24\" started-at=\"2018-06-13T12:30:21Z\" finished-at=\"2018-06-13T12:30:21Z\">\n\t\t  <reporter-output>\n\t\t  </reporter-output>\n\t\t</test-method> <!-- beforeMethod -->\n\t\t<test-method status=\"PASS\" signature=\"test()[pri:0, instance:com.mtfilemedia.ifive.testscripts.UIHTML5TC10SettingSupportUI@7e0e6aa2]\" name=\"test\" duration-ms=\"19520\" started-at=\"2018-06-13T12:30:21Z\" finished-at=\"2018-06-13T12:30:40Z\">\n\t\t</test-method> <!-- test -->\n\t\t<test-method status=\"PASS\" signature=\"afterMethod()[pri:0, instance:com.mtfilemedia.ifive.testscripts.UIHTML5TC10SettingSupportUI@7e0e6aa2]\" name=\"afterMethod\" is-config=\"true\" duration-ms=\"172\" started-at=\"2018-06-13T12:30:41Z\" finished-at=\"2018-06-13T12:30:42Z\">\n\t\t  <reporter-output>\n\t\t  </reporter-output>\n\t\t</test-method> <!-- afterMethod -->\n\t\t<test-method status=\"PASS\" signature=\"testTearDown()[pri:0, instance:com.mtfilemedia.ifive.testscripts.UIHTML5TC10SettingSupportUI@7e0e6aa2]\" name=\"testTearDown\" is-config=\"true\" duration-ms=\"2\" started-at=\"2018-06-13T12:30:42Z\" finished-at=\"2018-06-13T12:30:42Z\">\n\t\t  <reporter-output>\n\t\t  </reporter-output>\n\t\t</test-method> <!-- testTearDown -->\n\t  </class> <!-- com.mtfilemedia.ifive.testscripts.UIHTML5TC10SettingSupportUI -->\n\t</test>\n  </suite>\n</testng-results>",
           "tags" => [
        [0] "multiline"
    ],
           "host" => "castrovel.local",
       "@version" => "1",
     "@timestamp" => 2018-07-18T17:34:04.697Z,
           "path" => "/Users/yaauie/src/elastic/discuss-scratch/140291-xml-xpath/input.xml",
           "type" => "xml",
    "test-method" => [
        [0] "<test-method duration-ms=\"14\" finished-at=\"2018-06-13T12:30:21Z\" is-config=\"true\" name=\"testSetup\" signature=\"testSetup()[pri:0, instance:com.mtfilemedia.ifive.testscripts.UIHTML5TC10SettingSupportUI@7e0e6aa2]\" started-at=\"2018-06-13T12:30:21Z\" status=\"PASS\">\n\t\t  <reporter-output>\n\t\t  </reporter-output>\n\t\t</test-method>",
        [1] "<test-method duration-ms=\"24\" finished-at=\"2018-06-13T12:30:21Z\" is-config=\"true\" name=\"beforeMethod\" signature=\"beforeMethod()[pri:0, instance:com.mtfilemedia.ifive.testscripts.UIHTML5TC10SettingSupportUI@7e0e6aa2]\" started-at=\"2018-06-13T12:30:21Z\" status=\"PASS\">\n\t\t  <reporter-output>\n\t\t  </reporter-output>\n\t\t</test-method>",
        [2] "<test-method duration-ms=\"19520\" finished-at=\"2018-06-13T12:30:40Z\" name=\"test\" signature=\"test()[pri:0, instance:com.mtfilemedia.ifive.testscripts.UIHTML5TC10SettingSupportUI@7e0e6aa2]\" started-at=\"2018-06-13T12:30:21Z\" status=\"PASS\">\n\t\t</test-method>",
        [3] "<test-method duration-ms=\"172\" finished-at=\"2018-06-13T12:30:42Z\" is-config=\"true\" name=\"afterMethod\" signature=\"afterMethod()[pri:0, instance:com.mtfilemedia.ifive.testscripts.UIHTML5TC10SettingSupportUI@7e0e6aa2]\" started-at=\"2018-06-13T12:30:41Z\" status=\"PASS\">\n\t\t  <reporter-output>\n\t\t  </reporter-output>\n\t\t</test-method>",
        [4] "<test-method duration-ms=\"2\" finished-at=\"2018-06-13T12:30:42Z\" is-config=\"true\" name=\"testTearDown\" signature=\"testTearDown()[pri:0, instance:com.mtfilemedia.ifive.testscripts.UIHTML5TC10SettingSupportUI@7e0e6aa2]\" started-at=\"2018-06-13T12:30:42Z\" status=\"PASS\">\n\t\t  <reporter-output>\n\t\t  </reporter-output>\n\t\t</test-method>"
    ],
    "test-status" => [
        [0] "PASS"
    ]
}

yaauie · July 18, 2018, 6:01pm

If you want to split this event into one per test-method, that would actually be pretty easy to do from here:

filter {
  xml {
    remove_namespaces => true
    source => "message"
    store_xml => false
    xpath => [
      # only extract the test-method
      "/testng-results/suite/test/class/test-method", "test-method"
    ]
  }

  split {
    field => "test-method"
    remove_field => "message" # the whole message isn't particularly applicable to individual events after the split
  }

  xml {
    # expand test-method XML into an object in-place
    source => "test-method"
    target => "test-method"
  }
}

caveat: if your Elasticsearch index already has a string-type test-method, you'll need to expand the XML into a different location or else Elasticsearch will reject the insert.

satendr · July 19, 2018, 11:24am

Thanks a lot!!! Its working

system · August 16, 2018, 11:24am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.