Parsing nessus XML in Logstash

[magnusbaeck (Magnus Bäck)]

If you run Logstash form your terminal you'll get output from the stdout output to stdout, i.e. in your terminal.

And yes, this is what I'm looking for (message truncated):

{
          "host" => "debian",
          "tags" => [
        [0] "multiline",
        [1] "multiline_codec_max_lines_reached",
        [2] "nessus"
    ],
       "message" => "<?xml version=\"1.0\" ?>\n<NessusClientData_v2>\n<Policy><policyName>PCI Quarterly External Scan</policyName>\n<Preferences><ServerPreferences><preference><name>max_simult_tcp_sessions</name>\n<value>unlimited</value>\n</preference>\n<preference><name>use_mac_addr</name>\n<value>no</value>\n</preference>\n<preference>...",
          "type" => "nessus",
          "path" => "/home/najmi/nessus/alogolia_wk0wlr.nessus",
      "@version" => "1",
    "@timestamp" => 2018-02-23T13:36:04.496Z
}

Judging by the multiline_codec_max_lines_reached tag you'll have to raise its max_lines option (and maybe also max_bytes).

[wdaburu]

Thanks a lot @magnusbaeck your analysis is correct. It worked, logstash able to process the xml by adding those two line max_lines and max_bytes. Below are my current config file, i don't know if my max_lines parameter are overkill or not

input {
  file {
    path => "/home/najmi/nessus/*"
    sincedb_path => "/dev/null"
    start_position => "beginning"
    codec => multiline {
      pattern => "<Report |</NessusClientData_v2>"
      auto_flush_interval => 1
      negate => "true"
      what => "previous"
      max_lines => 1000000000
      max_bytes => "50 MiB"

    }
      tags => "nessus"
      type => "nessus"
  }
}

filter {
  ##interpret the message as XML
    if [type] == "nessus" {
        xml {
            source => "message"
            store_xml => "false"
            force_array => "false"
            xpath => ["/Report/@name", report_name]
            xpath => ["/Report/ReportHost/ReportItem/@pluginName", plugin_name]
            xpath => ["/Report/ReportHost/ReportItem/@pluginID", plugin_id]
            xpath => ["/Report/ReportHost/ReportItem/@severity", risk_score]
            xpath => ["/Report/ReportHost/ReportItem/@port", port]
            xpath => ["/Report/ReportHost/HostProperties/tag[@name='HOST_START']/text()", report_host_start]
            xpath => ["/Report/ReportHost/HostProperties/tag[@name='HOST_END']/text()", report_host_end]
        }
        mutate {
          remove_field => ["message"]
          convert => {
              "risk_score" => "integer"
          }

        }
        date {
            match => ["report_host_start", "EEE MMM dd HH:mm:ss yyyy"]
            target => "report_host_start"
            locale => "en_US"
        }
        date {
            match => ["report_host_end", "EEE MMM dd HH:mm:ss yyyy"]
            target => "report_host_end"
            locale => "en_US"
        }
    }
}

output {
      elasticsearch { 
        hosts => ["192.168.1.152:9200"]
        index => "nessus-data-%{+YYYY.MM.dd}"
        user => elastic
        password => password
      }
      stdout { codec => rubydebug }
}

By the way another question not related to the parsing, why is my data count is showing only 1 while the data contain more than that, for example the risk_score field for "0" value have 179 data, when i try to visualize it only show 1 count.

Hovering my mouse on the bar shows 179 occurrence

Visualizing those field shows only 1 count each.

[magnusbaeck (Magnus Bäck)]

Not knowing what the data looks like I don't have anything useful to say regarding your last question. Re the previous question of why you're only seeing 179 counts of zero it might be because the quick analysis you get when you click on a field it's based on the 500 most recent events.