Add new elements in XML array

Elastic Stack Logstash
1

Hi all !

I have the following XML file:

<?xml version="1.0" encoding="ISO-8859-1"?>
<?xml-stylesheet type="text/xsl" href="/content/Results.xsl"?>
<results>


    <testcases file="tests/testcases.xml">

        <testcase id="1">
            <description1>website:orange.fr - website_url:https://www.orange.fr - website_port:443 - website_localisation:France - network:SITES_DISTANTS</description1>
            <verifypositive>
                <assert>Clients Orange</assert>
                <success>true</success>
            </verifypositive>
            <verifyresponsecode-success>true</verifyresponsecode-success>
            <verifyresponsecode-message>Passed HTTP Response Code Verification</verifyresponsecode-message>
            <success>true</success>
            <result-message>TEST CASE PASSED</result-message>
            <responsetime>0.147</responsetime>
        </testcase>

        <testcase id="2">
            <description1>website:google.com - website_url:https://www.google.com - website_port:443 - website_localisation:Etats-Unis - network:SITES_DISTANTS</description1>
            <verifypositive>
                <assert>Google</assert>
                <success>true</success>
            </verifypositive>
            <verifyresponsecode-success>true</verifyresponsecode-success>
            <verifyresponsecode-message>Passed HTTP Response Code Verification</verifyresponsecode-message>
            <success>true</success>
            <result-message>TEST CASE PASSED</result-message>
            <responsetime>0.171</responsetime>
        </testcase>
</testcases>

    <test-summary>
        <start-time>Sat 13 Jan 2018, 20:19:12</start-time>
        <start-seconds>73152</start-seconds>
        <start-date-time>2018-01-13T20:19:12</start-date-time>
        <total-run-time>2.346</total-run-time>
        <total-response-time>2.163</total-response-time>
        <test-cases-run>12</test-cases-run>
        <test-cases-passed>12</test-cases-passed>
        <test-cases-failed>0</test-cases-failed>
        <verifications-passed>36</verifications-passed>
        <verifications-failed>0</verifications-failed>
        <assertion-skips>0</assertion-skips>
        <average-response-time>0.18</average-response-time>
        <max-response-time>0.277</max-response-time>
        <min-response-time>0.136</min-response-time>
        <execution-aborted>false</execution-aborted>
        <test-file-name>testcases</test-file-name>
    </test-summary>

</results>

And I need to create few elements from one element in all occurence

<description1>website:google.com - website_url:https://www.google.com - website_port:443 - website_localisation:Etats-Unis - network:SITES_DISTANTS</description1>

To have that:

<?xml version="1.0" encoding="ISO-8859-1"?>
<?xml-stylesheet type="text/xsl" href="/content/Results.xsl"?>
<results>


    <testcases file="tests/testcases.xml">

        <testcase id="1">
            <description1>website:orange.fr - website_url:https://www.orange.fr - website_port:443 - website_localisation:France - network:SITES_DISTANTS</description1>
            <website>orange.fr </website>
            <website_url>https://www.orange.fr</website_url>
            <website_port>443</website_port>
            <website_localisation>France</website_localisation>
            <network>SITES_DISTANTS</network>
            <verifypositive>
                <assert>Clients Orange</assert>
                <success>true</success>
            </verifypositive>
            <verifyresponsecode-success>true</verifyresponsecode-success>
            <verifyresponsecode-message>Passed HTTP Response Code Verification</verifyresponsecode-message>
            <success>true</success>
            <result-message>TEST CASE PASSED</result-message>
            <responsetime>0.147</responsetime>
        </testcase>

        <testcase id="2">
            <description1>website:google.com - website_url:https://www.google.com - website_port:443 - website_localisation:Etats-Unis - network:SITES_DISTANTS</description1>
            <website>google.com</website>
            <website_url>:https://www.google.com</website_url>
            <website_port>443</website_port>
            <website_localisation>Etats-Unis</website_localisation>
            <network>SITES_DISTANTS</network>
            <verifypositive>
                <assert>Google</assert>
                <success>true</success>
            </verifypositive>
            <verifyresponsecode-success>true</verifyresponsecode-success>
            <verifyresponsecode-message>Passed HTTP Response Code Verification</verifyresponsecode-message>
            <success>true</success>
            <result-message>TEST CASE PASSED</result-message>
            <responsetime>0.171</responsetime>
        </testcase>
</testcases>

    <test-summary>
        <start-time>Sat 13 Jan 2018, 20:19:12</start-time>
        <start-seconds>73152</start-seconds>
        <start-date-time>2018-01-13T20:19:12</start-date-time>
        <total-run-time>2.346</total-run-time>
        <total-response-time>2.163</total-response-time>
        <test-cases-run>12</test-cases-run>
        <test-cases-passed>12</test-cases-passed>
        <test-cases-failed>0</test-cases-failed>
        <verifications-passed>36</verifications-passed>
        <verifications-failed>0</verifications-failed>
        <assertion-skips>0</assertion-skips>
        <average-response-time>0.18</average-response-time>
        <max-response-time>0.277</max-response-time>
        <min-response-time>0.136</min-response-time>
        <execution-aborted>false</execution-aborted>
        <test-file-name>testcases</test-file-name>
    </test-summary>

</results>

Do you think it is possible ?

Actually, I don't know how to beggining but I have that:

input {
  beats {
    port => 7001
  }
}

filter {
  xml {
    source => "message"
    target => "parsed"
  }
}

output {
    stdout { codec => rubydebug }
    elasticsearch {
      hosts => ["192.168.0.11:9200"]
      index => "webinject-%{+YYYY.MM.dd}"
      template => "/etc/logstash/templates/webinject.json"
      template_name => "webinject"

    }
}

Thank You !!

2

Do you want an event per testcase? If no, the solution is hard because you don't know how many testcases there are to build a KV filter to work on.

Generally, this is the workflow or compute flow:
Use the split filter on the [testcases] field - this will yield X new events with the same structure.
Use the KV filter on the testcase][description1] field - delimiter => " - " separator => ":" with the target [testcase]

Something like this (I reduced the xml to make the output easier to read)...

input {
  generator {
    message => '<?xml version="1.0" encoding="ISO-8859-1"?>
<?xml-stylesheet type="text/xsl" href="/content/Results.xsl"?>
<results>
    <testcases file="tests/testcases.xml">
        <testcase id="1">
            <description1>website:orange.fr - website_url:https://www.orange.fr - website_port:443 - website_localisation:France - network:SITES_DISTANTS</description1>
            <success>true</success>
        </testcase>

        <testcase id="2">
            <description1>website:google.com - website_url:https://www.google.com - website_port:443 - website_localisation:Etats-Unis - network:SITES_DISTANTS</description1>
            <success>true</success>
        </testcase>
    </testcases>
    <test-summary>
        <start-time>Sat 13 Jan 2018, 20:19:12</start-time>
    </test-summary>

</results>'
    count => 1
  }
}
filter {
  xml {
   source => "message"
   target => "parsed"

  }
  split {
    field => "[parsed][testcases][0][testcase]"
    target => "[testcase]"
    remove_field => [ "[message]", "[parsed]" ]
  }
  mutate {
    rename => {
      "[testcase][success]" => "[testcase][successX]"
      "[testcase][description1]" => "[testcase][descriptionX]"
    }
    add_field => {
      "[testcase][success]" => "%{[testcase][successX][0]}"
      "[testcase][description1]" => "%{[testcase][descriptionX][0]}"
    }
    remove_field => [ "[testcase][successX]", "[testcase][descriptionX]" ]
  }
  kv {
    field_split => " - "
    value_split => ":"
    source => "[testcase][description1]"
    target => "[testcase][description]"
    remove_field => [ "[testcase][description1]" ]
  }
}

output {
  stdout { codec => rubydebug }
}

The above is the long-hand way of massaging the field names to get rid on the Array values.
The other method is to use XPath extractions. See my post here

Results:

{
      "testcase" => {
        "description" => {
                    "website_port" => "443",
            "website_localisation" => "France",
                         "network" => "SITES_DISTANTS",
                     "website_url" => "https://www.orange.fr",
                         "website" => "orange.fr"
        },
                 "id" => "1",
            "success" => "true"
    },
          "host" => "Elastics-MacBook-Pro.local",
      "sequence" => 0,
      "@version" => "1",
    "@timestamp" => 2018-01-17T14:52:29.651Z
}
{
      "testcase" => {
        "description" => {
                    "website_port" => "443",
            "website_localisation" => "Etats-Unis",
                         "network" => "SITES_DISTANTS",
                     "website_url" => "https://www.google.com",
                         "website" => "google.com"
        },
                 "id" => "2",
            "success" => "true"
    },
          "host" => "Elastics-MacBook-Pro.local",
      "sequence" => 0,
      "@version" => "1",
    "@timestamp" => 2018-01-17T14:52:29.651Z
}
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.