Is it possible to change output structure


#7

Nice, that works.

How am I able to split the array of reportitem, so i can get each array([0],[1],[2]) an own event, because if i index now, i just get one event.
The events should looks like this:

event[0]
Severity: 0
ip: 11.111.11
port: 0
HistPropertiesName: LastUnauthenticatedResults

event[1]
Severity: 3
ip: 11.111.11
port: 22
HistPropertiesName: LastUnauthenticatedResults


(Christian Dahlqvist) #8

Have you tried using the split filter on the reportitem field?


#9

Now I tried and it's perfect. Thx a lot guys :slight_smile:


#10

Is there a reason why you pick s. each_index?


#11

the reason why i choose s because it contains 3 elements and i want to iterate the loop for 3 times.
You need to mention the name of the array, where you can iterate with the size of the elements


#12

I have some more questions.

When i have a log, where are elements with the same name but different values, how can i get them indexed in the same index?
I.E.

<?xml version="1.0" ?>
<system>
<Report name="Scan">
<ReportHost ip="11.111.11.1"><HostProperties>
<tag pluginname="LastUnauthenticatedResults">1539<tag>
</HostProperties>
<ReportItem port="80" severity="5">
**<bid>1111</bid>**
**<bid>2222</bid>**
</ReportItem>
</ReportHost>
</Report>
</system>

I guess i need smth like

s.each_index { |k,j|
h = { 'portname' => p[k] , 'severity' => s[k] , 'ip' => i[0], 'pluginname' => n[0], 'bid'[j]=>[k] }
carr << h
}

#13

yes you need to iterate again, give me the sample output required for you


#14

this


#15

For this, again you get "bid" like array of objects and you need to iterate(bid array) within the loop( reportitem array)


#16

I work with ruby since yesterday. Have to search how it works.


#17

The input code as per your requirement is as follows,

input {
file {
path => "D:/xxxxx/ELKStack/sample.xml"
start_position => "beginning"
sincedb_path => "NUL"
codec => multiline {
pattern => ""
negate => "true"
what => "previous"
auto_flush_interval => 1
max_lines => 333333
}
}
}

filter {
xml {
source => "message"
target => "parsed"
store_xml => "false"
xpath => [
"/system/Report/ReportHost/@ip","ip",
"/system/Report/ReportHost/HostProperties/tag/@pluginname","pluginname",
"/system/Report/ReportHost/HostProperties/tag/text()","content",
"/system/Report/ReportHost/ReportItem/@port","portname",
"/system/Report/ReportHost/ReportItem/@severity","severity",
"/system/Report/ReportHost/ReportItem/bid/text()","bidvalue"
]
}

ruby {
code => "
i = event.get('ip')
n = event.get('pluginname')
p = event.get('portname')
s = event.get('severity')
b = event.get('bidvalue')
carr =
s.each_index { |k|
h = { 'portname' => p[k] , 'severity' => s[k] , 'ip' => i[0], 'pluginname' => n[0],'bid1' => b[0],'bid2' => b[1] }
carr << h
}
event.set('reportitem', carr) "
}
mutate {
remove_field => ["message","@version"]
}

}

output {
stdout {
codec => rubydebug
}
}

The output is as shown in below image,

Capture


Xml filter with keyvalue pair
XML import
#18

hope above solution help you, :slight_smile:


#19

the problem is, in my log the elementnames are exactly the same. they both called and i cant say
bid1=>b[0]
bid2=>b[1]


#20

Based on the xml file given by you, i have tested and it worked fine. Please check the output image attached above.

Please let me know if you have an unformatted xml file, if so, post them.


#21

I didn't expect this will work, but it does.
I search smth what works when I don't know how many bid elements I have in my logfile.


#22

Yup, probably your xml file is not in unique format.
The format is different with the bid elements, then we need to change the code,

if your xml file is in unique format there will not be any problem.


#23

Ah no, it's not unique. I try to build a script which will work above XML files which can be different a bit different to each other. But the differences aren't that big. At the moment I try to learn ruby a bit so I work with one file and try to applay it to others.


#24

My xml files looks like my sample but with multiple elements. The values can be different in every field.

<?xml version="1.0" ?>
<system>
<Report name="Scan">
<ReportHost ip="11.111.11.1"><HostProperties>
<tag pluginname="LastUnauthenticatedResults">1539<tag>
<tag pluginnname="Scan">false</tag>
</HostProperties>
<ReportItem port="0" severity="0">
</ReportItem>
<ReportItem port="22" severity="3">
</ReportItem>
<ReportItem port="80" severity="5">
</ReportItem>
</ReportHost>
<ReportHost ip="128.98.11.9"><HostProperties>
<tag pluginname="os">linux<tag>
<tag pluginname="cpe">linux_kernel</tag>
</HostProperties>
<ReportItem port="17" severity="3">
</ReportItem>
<ReportItem port="22" severity="3">
<bid>1111</bid>
<bid>2222</bid>
</ReportItem>
<ReportItem port="80" severity="7">
<bid>1787</bid>
<cve>CVE-2008-5161</cve>
<cve>CVE-2007-6713</cve>
<solution>n/a</solution>
</ReportItem>
</ReportHost>
</Report>
</system>

And my files have about 300000 lines where they should work...


#25

it sounds like a good idea, try it.


(system) closed #26

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.