JSON filter plugin explain example please


#1

One moment, i need to correct my post. I converted the file into json. Is it easier to work with xml or json files or is it the same difficulty?


(Christian Dahlqvist) #2

That is not JSON, that is XML. Therefore use the xml filter plugin.


#3

I got JSON file looks like this:

{
"NessusClientData_v2": {
"Policy": {
"policyName": "Arena Standard",
"Preferences": {
"ServerPreferences": {
"preference": [
{ "name": "max_simult_tcp_sessions" },
{
"name": "use_mac_addr",
"value": "no"
},
{
"name": "sc_version",
"value": "5.6.1"
},
......
{
"-port": "80",
"-svc_name": "www",
"-protocol": "tcp",
"-severity": "0",
"-pluginID": "11219",
"-pluginName": "bli",
"-pluginFamily": "bla",
"description": "blub",
"fname": "123
},
{
"-port": "443",
"-svc_name": "www",
"-protocol": "tcp",
"-severity": "0",
"-pluginID": "11219",
"-pluginName": "bliii",
"-pluginFamily": "blaaa",
"description": "blub"
}
]
}
]
}
}
}

As result i want every block in one event:

port: "80"
svc_name: "www"
protocol: "tcp"
severity: "0"
pluginID: "11219"
pluginName": "bli"
pluginFamily": "bla"
description": "blub"
fname": "123"

I tried to work with multilines. I got one event with all entries of the file and i didn't know how to split this up to get more events. Can someone give a small example or link where it's explained a bit?

My logstash configs looks like this atm:

input {
  file {
        path => "/home/vagrant/test.json"
        start_position => "beginning"
        type => "123"
  }
}

filter {
  if [message] == "123" {
     json {
       source => "message"
       skip_on_invalid_json => "true"

    }
  }
}


output {
  elasticsearch {
    index => "123"
    hosts => "127.0.0.1"
  }
}

#4

I would do it with the ruby plugin filter.

Once you have your entire json file in one single event, you have to loop on your items in your ruby script and put them in separate event array fields.


#5

When i take a look at your link, i don't know how to start to get my result. Is there somewhere an example where i can learn it with my issue?


#6

You can begin with that http://bfy.tw/KNhK and if you need help on your specific implementation we'll see.

By the way, I'm not sure your json file is valid because your block elements don't seem to have a name ... Are they in an array structure ? If not, I don't know how you gonna loop on them :thinking:


(Christian Dahlqvist) #7

Use a multiline codec to gather the full event into one message, then apply a json filter. To split this into multiple events, you might be able to use a split filter on the preference field. If that does not work for you, you may need a ruby filter.


#8

originally it looks like this

"ReportItem": [
            {
              "-port": "0",
              "-svc_name": "general",
              "-protocol": "tcp",
              "-severity": "0",
              "-pluginID": "117886",
              "-pluginName": "Local Checks Not Enabled (info)",
              "-pluginFamily": "Settings",
              "description": "bla",
              "fname": "bla",
              "plugin_modification_date": "bla",
              "plugin_name": "bla",
              "plugin_publication_date": "2016/10/02",
              "plugin_type": "summary",
              "risk_factor": "None",
              "script_version": "1.2",
              "solution": "n/a",
              "synopsis": "bla",
              "plugin_output": "bla"
},
            {
              "-port": "0",
              "-svc_name": "general",
              "-protocol": "tcp",
              "-severity": "0",
              "-pluginID": "19506",
             .....

#9

ok i will try that, thx


(system) #11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.