Is it possible to change output structure

Elastic Stack Logstash
1

Hello,
I got an output structure like that:

{
              "path" => "/home/data/test.xml",
          "@version" => "1",
        "@timestamp" => 2018-10-24T08:49:15.480Z,
    "Properties" => {
               "tag" => [
            [0] {
                   "pluginname" => "LastUnauthenticatedResults",
                      "content" => "1539"
            },
            [1] {
                   "pluginname" => "Scan",
                      "content" => "false"
            }
        ],
        "ReportItem" => [
            [0] {
                          "port" => "0",
                      "severity" => "0",
            },
            [1] {
                        "port" => "22"
                    "severity" => "3"
            },
            [2] {
                        "port" => "80"
                    "severity" => "5"
           }
        ]
    },
                "ip" => "11.111.11.1"
}

I am not able to index this structure.
Is it possible to get output like that

{
          
        "ReportItem" => [
            [0] {
                          "port" => "0",
                      "severity" => "0",
                    "pluginname" => "LastUnauthenticatedResults",
                       "content" => "1539"
                            "ip" => "11.111.11.1"
            },
            [1] {
                        "port" => "22"
                    "severity" => "3"
                  "pluginname" => "LastUnauthenticatedResults",
                     "content" => "1539"
                          "ip" => "11.111.11.1"
            },

            [2] {
                        "port" => "80"
                    "severity" => "5"
                  "pluginname" => "Scan",
                     "content" => "false"
                          "ip" => "11.111.11.1"
            }
        ]
   }

What i need to use if that's possible?

2

Yup, it is possible by writing the ruby code, but if you could provide the input code, it is easy to provide the solution.

3

Sure, my input code is this:

input {
  file {
    path => "/home/data/test.xml"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => multiline {
      pattern => "</system>"
      what => "next"
      negate => true
      max_lines => 333333
    }
  }
}

filter {
  xml {
    source => "message"
    store_xml => false
    xpath => ["/system/Report/ReportHost","ReportHost"]    
  }
  mutate {
    remove_field => ["message","@version"]
  }
  split {
    field => "ReportHost"
  }
  xml {
    source => "ReportHost"
    target => "[xml_content]"
    force_array => false
  }
  ruby {
    code => '
      event.get("[xml_content]").each do |key, value|
        event.set(key, value)
      end
    '
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

Do you also need my input data structure?

4

Also provide the sample xml file also

5

my xml sample

<?xml version="1.0" ?>
<system>
<Report name="Scan">
<ReportHost ip="11.111.11.1"><HostProperties>
<tag pluginname="LastUnauthenticatedResults">1539<tag>
<tag pluginnname="Scan">false</tag>
</HostProperties>
<ReportItem port="0" severity="0">
</ReportItem>
<ReportItem port="22" severity="3">
</ReportItem>
<ReportItem port="80" severity="5">
</ReportItem>
</ReportHost>
</Report>
</system>
6

The xml file which you have posted has an error, didnot close the tag
<tag pluginname="LastUnauthenticatedResults">1539</tag>

The input code is as follows as per your requirement,

input {
file {
path => "D:/xxxxx/ELKStack/sample.xml"
start_position => "beginning"
sincedb_path => "NUL"
codec => multiline {
pattern => ""
negate => "true"
what => "previous"
auto_flush_interval => 1
max_lines => 333333
}
}
}

filter {
xml {
source => "message"
target => "parsed"
store_xml => "false"
xpath => [
"/system/Report/ReportHost/@ip","ip",
"/system/Report/ReportHost/HostProperties/tag/@pluginname","pluginname",
"/system/Report/ReportHost/HostProperties/tag/text()","content",
"/system/Report/ReportHost/ReportItem/@port","portname",
"/system/Report/ReportHost/ReportItem/@severity","severity"
]
}

ruby {
code => "
i = event.get('ip')
n = event.get('pluginname')
p = event.get('portname')
s = event.get('severity')
carr =
s.each_index { |k|
h = { 'portname' => p[k] , 'severity' => s[k] , 'ip' => i[0], 'pluginname' => n[0] }
carr << h
}
event.set('reportitem', carr) "
}
mutate {
remove_field => ["message","@version"]
}

}

output {
stdout {
codec => rubydebug
}
}

the output is as shown in the image below

Capture

2 Likes
7

Nice, that works.

How am I able to split the array of reportitem, so i can get each array([0],[1],[2]) an own event, because if i index now, i just get one event.
The events should looks like this:

event[0]
Severity: 0
ip: 11.111.11
port: 0
HistPropertiesName: LastUnauthenticatedResults

event[1]
Severity: 3
ip: 11.111.11
port: 22
HistPropertiesName: LastUnauthenticatedResults

8

Have you tried using the split filter on the reportitem field?

1 Like
9

Now I tried and it's perfect. Thx a lot guys :slight_smile:

10

Is there a reason why you pick s. each_index?

11

the reason why i choose s because it contains 3 elements and i want to iterate the loop for 3 times.
You need to mention the name of the array, where you can iterate with the size of the elements

1 Like
12

I have some more questions.

When i have a log, where are elements with the same name but different values, how can i get them indexed in the same index?
I.E.

<?xml version="1.0" ?>
<system>
<Report name="Scan">
<ReportHost ip="11.111.11.1"><HostProperties>
<tag pluginname="LastUnauthenticatedResults">1539<tag>
</HostProperties>
<ReportItem port="80" severity="5">
**<bid>1111</bid>**
**<bid>2222</bid>**
</ReportItem>
</ReportHost>
</Report>
</system>

I guess i need smth like

s.each_index { |k,j|
h = { 'portname' => p[k] , 'severity' => s[k] , 'ip' => i[0], 'pluginname' => n[0], 'bid'[j]=>[k] }
carr << h
}
13

yes you need to iterate again, give me the sample output required for you

14

this

15

For this, again you get "bid" like array of objects and you need to iterate(bid array) within the loop( reportitem array)

16

I work with ruby since yesterday. Have to search how it works.

17

The input code as per your requirement is as follows,

input {
file {
path => "D:/xxxxx/ELKStack/sample.xml"
start_position => "beginning"
sincedb_path => "NUL"
codec => multiline {
pattern => ""
negate => "true"
what => "previous"
auto_flush_interval => 1
max_lines => 333333
}
}
}

filter {
xml {
source => "message"
target => "parsed"
store_xml => "false"
xpath => [
"/system/Report/ReportHost/@ip","ip",
"/system/Report/ReportHost/HostProperties/tag/@pluginname","pluginname",
"/system/Report/ReportHost/HostProperties/tag/text()","content",
"/system/Report/ReportHost/ReportItem/@port","portname",
"/system/Report/ReportHost/ReportItem/@severity","severity",
"/system/Report/ReportHost/ReportItem/bid/text()","bidvalue"
]
}

ruby {
code => "
i = event.get('ip')
n = event.get('pluginname')
p = event.get('portname')
s = event.get('severity')
b = event.get('bidvalue')
carr =
s.each_index { |k|
h = { 'portname' => p[k] , 'severity' => s[k] , 'ip' => i[0], 'pluginname' => n[0],'bid1' => b[0],'bid2' => b[1] }
carr << h
}
event.set('reportitem', carr) "
}
mutate {
remove_field => ["message","@version"]
}

}

output {
stdout {
codec => rubydebug
}
}

The output is as shown in below image,

Capture

18

hope above solution help you, :slight_smile:

19

the problem is, in my log the elementnames are exactly the same. they both called and i cant say
bid1=>b[0]
bid2=>b[1]

20

Based on the xml file given by you, i have tested and it worked fine. Please check the output image attached above.

Please let me know if you have an unformatted xml file, if so, post them.