Is it possible to change output structure


#1

Hello,
I got an output structure like that:

{
              "path" => "/home/data/test.xml",
          "@version" => "1",
        "@timestamp" => 2018-10-24T08:49:15.480Z,
    "Properties" => {
               "tag" => [
            [0] {
                   "pluginname" => "LastUnauthenticatedResults",
                      "content" => "1539"
            },
            [1] {
                   "pluginname" => "Scan",
                      "content" => "false"
            }
        ],
        "ReportItem" => [
            [0] {
                          "port" => "0",
                      "severity" => "0",
            },
            [1] {
                        "port" => "22"
                    "severity" => "3"
            },
            [2] {
                        "port" => "80"
                    "severity" => "5"
           }
        ]
    },
                "ip" => "11.111.11.1"
}

I am not able to index this structure.
Is it possible to get output like that

{
          
        "ReportItem" => [
            [0] {
                          "port" => "0",
                      "severity" => "0",
                    "pluginname" => "LastUnauthenticatedResults",
                       "content" => "1539"
                            "ip" => "11.111.11.1"
            },
            [1] {
                        "port" => "22"
                    "severity" => "3"
                  "pluginname" => "LastUnauthenticatedResults",
                     "content" => "1539"
                          "ip" => "11.111.11.1"
            },

            [2] {
                        "port" => "80"
                    "severity" => "5"
                  "pluginname" => "Scan",
                     "content" => "false"
                          "ip" => "11.111.11.1"
            }
        ]
   }

What i need to use if that's possible?


#2

Yup, it is possible by writing the ruby code, but if you could provide the input code, it is easy to provide the solution.


#3

Sure, my input code is this:

input {
  file {
    path => "/home/data/test.xml"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => multiline {
      pattern => "</system>"
      what => "next"
      negate => true
      max_lines => 333333
    }
  }
}

filter {
  xml {
    source => "message"
    store_xml => false
    xpath => ["/system/Report/ReportHost","ReportHost"]    
  }
  mutate {
    remove_field => ["message","@version"]
  }
  split {
    field => "ReportHost"
  }
  xml {
    source => "ReportHost"
    target => "[xml_content]"
    force_array => false
  }
  ruby {
    code => '
      event.get("[xml_content]").each do |key, value|
        event.set(key, value)
      end
    '
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

Do you also need my input data structure?


#4

Also provide the sample xml file also


#5

my xml sample

<?xml version="1.0" ?>
<system>
<Report name="Scan">
<ReportHost ip="11.111.11.1"><HostProperties>
<tag pluginname="LastUnauthenticatedResults">1539<tag>
<tag pluginnname="Scan">false</tag>
</HostProperties>
<ReportItem port="0" severity="0">
</ReportItem>
<ReportItem port="22" severity="3">
</ReportItem>
<ReportItem port="80" severity="5">
</ReportItem>
</ReportHost>
</Report>
</system>

#6

The xml file which you have posted has an error, didnot close the tag
<tag pluginname="LastUnauthenticatedResults">1539</tag>

The input code is as follows as per your requirement,

input {
file {
path => "D:/xxxxx/ELKStack/sample.xml"
start_position => "beginning"
sincedb_path => "NUL"
codec => multiline {
pattern => ""
negate => "true"
what => "previous"
auto_flush_interval => 1
max_lines => 333333
}
}
}

filter {
xml {
source => "message"
target => "parsed"
store_xml => "false"
xpath => [
"/system/Report/ReportHost/@ip","ip",
"/system/Report/ReportHost/HostProperties/tag/@pluginname","pluginname",
"/system/Report/ReportHost/HostProperties/tag/text()","content",
"/system/Report/ReportHost/ReportItem/@port","portname",
"/system/Report/ReportHost/ReportItem/@severity","severity"
]
}

ruby {
code => "
i = event.get('ip')
n = event.get('pluginname')
p = event.get('portname')
s = event.get('severity')
carr = []
s.each_index { |k|
h = { 'portname' => p[k] , 'severity' => s[k] , 'ip' => i[0], 'pluginname' => n[0] }
carr << h
}
event.set('reportitem', carr) "
}
mutate {
remove_field => ["message","@version"]
}

}

output {
stdout {
codec => rubydebug
}
}

the output is as shown in the image below

Capture


Trouble getting XML into usable format
Handle nested xml file with logstash and ruby script
Help parsing Xml File
#7

Nice, that works.

How am I able to split the array of reportitem, so i can get each array([0],[1],[2]) an own event, because if i index now, i just get one event.
The events should looks like this:

event[0]
Severity: 0
ip: 11.111.11
port: 0
HistPropertiesName: LastUnauthenticatedResults

event[1]
Severity: 3
ip: 11.111.11
port: 22
HistPropertiesName: LastUnauthenticatedResults


(Christian Dahlqvist) #8

Have you tried using the split filter on the reportitem field?


#9

Now I tried and it's perfect. Thx a lot guys :slight_smile:


#10

Is there a reason why you pick s. each_index?


#11

the reason why i choose s because it contains 3 elements and i want to iterate the loop for 3 times.
You need to mention the name of the array, where you can iterate with the size of the elements


#12

I have some more questions.

When i have a log, where are elements with the same name but different values, how can i get them indexed in the same index?
I.E.

<?xml version="1.0" ?>
<system>
<Report name="Scan">
<ReportHost ip="11.111.11.1"><HostProperties>
<tag pluginname="LastUnauthenticatedResults">1539<tag>
</HostProperties>
<ReportItem port="80" severity="5">
**<bid>1111</bid>**
**<bid>2222</bid>**
</ReportItem>
</ReportHost>
</Report>
</system>

I guess i need smth like

s.each_index { |k,j|
h = { 'portname' => p[k] , 'severity' => s[k] , 'ip' => i[0], 'pluginname' => n[0], 'bid'[j]=>[k] }
carr << h
}

#13

yes you need to iterate again, give me the sample output required for you


#14

this


#15

For this, again you get "bid" like array of objects and you need to iterate(bid array) within the loop( reportitem array)


#16

I work with ruby since yesterday. Have to search how it works.


#17

The input code as per your requirement is as follows,

input {
file {
path => "D:/xxxxx/ELKStack/sample.xml"
start_position => "beginning"
sincedb_path => "NUL"
codec => multiline {
pattern => ""
negate => "true"
what => "previous"
auto_flush_interval => 1
max_lines => 333333
}
}
}

filter {
xml {
source => "message"
target => "parsed"
store_xml => "false"
xpath => [
"/system/Report/ReportHost/@ip","ip",
"/system/Report/ReportHost/HostProperties/tag/@pluginname","pluginname",
"/system/Report/ReportHost/HostProperties/tag/text()","content",
"/system/Report/ReportHost/ReportItem/@port","portname",
"/system/Report/ReportHost/ReportItem/@severity","severity",
"/system/Report/ReportHost/ReportItem/bid/text()","bidvalue"
]
}

ruby {
code => "
i = event.get('ip')
n = event.get('pluginname')
p = event.get('portname')
s = event.get('severity')
b = event.get('bidvalue')
carr = []
s.each_index { |k|
h = { 'portname' => p[k] , 'severity' => s[k] , 'ip' => i[0], 'pluginname' => n[0],'bid1' => b[0],'bid2' => b[1] }
carr << h
}
event.set('reportitem', carr) "
}
mutate {
remove_field => ["message","@version"]
}

}

output {
stdout {
codec => rubydebug
}
}

The output is as shown in below image,

Capture


Xml filter with keyvalue pair
XML import
#18

hope above solution help you, :slight_smile:


#19

the problem is, in my log the elementnames are exactly the same. they both called and i cant say
bid1=>b[0]
bid2=>b[1]


#20

Based on the xml file given by you, i have tested and it worked fine. Please check the output image attached above.

Please let me know if you have an unformatted xml file, if so, post them.