Hello All,
I am trying to index into Logstash, a .csv (comma delimited) file generated from my Nessus scanner. However, I am receiving error related most probably to parsing problems, because in the Excel file, there are fields with text which contains special characters - \n, \t, "",n/a, etc. When I remove the fields/columnns the rest of the text is indexed successfully. I tried to mutate them, but without success. I am providing you guys with my logstash.conf file and sample from my .csv .How can I successfully parse the file?
Error:
[2020-03-10T09:29:51,119][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-03-10T09:29:51,903][WARN ][logstash.filters.csv ][main] Error parsing csv {:field=>"message", :source=>""None","XXXXXXXX","HTTP Server Type and Version","This plugin attempts to determine the type and the version of the", :exception=>#<CSV::MalformedCSVError: Unclosed quoted field on line 1.>}
[2020-03-10T09:29:51,940][WARN ][logstash.filters.csv ][main] Error parsing csv {:field=>"message", :source=>" remote web server.","n/a"\r", :exception=>#<CSV::MalformedCSVError: Illegal quoting in line 1.>}
[2020-03-10T09:29:51,945][WARN ][logstash.filters.csv ][main] Error parsing csv {:field=>"message", :source=>""None","XXXXXXXX","HTTP Server Type and Version","This plugin attempts to determine the type and the version of the", :exception=>#<CSV::MalformedCSVError: Unclosed quoted field on line 1.>}
[2020-03-10T09:29:51,949][WARN ][logstash.filters.csv ][main] Error parsing csv {:field=>"message", :source=>""None","XXXXXXXX","ICMP Timestamp Request Remote Date Disclosure","The remote host answers to an ICMP timestamp request. This allows an", :exception=>#<CSV::MalformedCSVError: Unclosed quoted field on line 1.>}
[2020-03-10T09:29:51,964][WARN ][logstash.filters.csv ][main] Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
Blockquote
input {
file {
path => "/opt/VulnWhisperer/data/nessus/My Scans/CORE*"
type => "csv"
mode => "read"
start_position => "beginning"
sincedb_path => "/dev/null"
file_completed_action => "delete"
tags => "nessus"
}
}
filter {
if "nessus" in [tags] or "tenable" in [tags] {
# Drop the header column
#if [message] =~ "^Plugin ID" { drop {} }
csv {
columns => ["Risk", "Host", "Name", "Description", "Solution"]
separator => ","
source => "message"
}
mutate {
gsub => ["message", "n/a", "" ]
}
ruby {
code => "if event.get('description')
event.set('description', event.get('description').gsub(92.chr + 'n', 10.chr).gsub(92.chr + 'r', 13.chr))
end
if event.get('synopsis')
event.set('synopsis', event.get('synopsis').gsub(92.chr + 'n', 10.chr).gsub(92.chr + 'r', 13.chr))
end
if event.get('solution')
event.set('solution', event.get('solution').gsub(92.chr + 'n', 10.chr).gsub(92.chr + 'r', 13.chr))
end
if event.get('see_also')
event.set('see_also', event.get('see_also').gsub(92.chr + 'n', 10.chr).gsub(92.chr + 'r', 13.chr))
end
if event.get('plugin_output')
event.set('plugin_output', event.get('plugin_output').gsub(92.chr + 'n', 10.chr).gsub(92.chr + 'r', 13.chr))
end"
}
#If using filebeats as your source, you will need to replace the "path" field to "source"
# Remove when scan name is included in event (current method is error prone)
grok {
match => { "path" => "(?<scan_name>[a-zA-Z0-9_.\-]+)_%{INT:scan_id}_%{INT:history_id}_%{INT:last_updated}.csv$" }
tag_on_failure => []
}
# TODO remove when @timestamp is included in event
date {
match => [ "last_updated", "UNIX" ]
target => "@timestamp"
remove_field => ["last_updated"]
}
output {
if "nessus" in [tags]{
stdout {
codec => dots
}
elasticsearch {
hosts => [ "192.168.83.137:9200" ]
index => "core-banking-vulnwhisperer-%{+YYYY.MM}"
}
}
}
Blockquote
Here are some samples from my .csv file:
Risk Host Name Description Solution
None XXXXXXXX HTTP Server Type and Version "This plugin attempts to determine the type and the version of the
remote web server." n/a
None XXXXXXXX HTTP Server Type and Version "This plugin attempts to determine the type and the version of the
remote web server." n/a
None XXXXXXXX ICMP Timestamp Request Remote Date Disclosure "The remote host answers to an ICMP timestamp request. This allows an
attacker to know the date that is set on the targeted machine, which
may assist an unauthenticated, remote attacker in defeating time-based
authentication protocols.
Timestamps returned from machines running Windows Vista / 7 / 2008 /
2008 R2 are deliberately incorrect, but usually within 1000 seconds of
the actual system time." "Filter out the ICMP timestamp requests (13), and the outgoing ICMP
timestamp replies (14)."
None XXXXXXXX Windows NetBIOS / SMB Remote Host Information Disclosure "The remote host is listening on UDP port 137 or TCP port 445, and
replies to NetBIOS nbtscan or SMB requests.
Note that this plugin gathers information to be used in other plugins,
but does not itself generate a report." n/a
None XXXXXXXX Ping the remote host "Nessus was able to determine if the remote host is alive using one or
more of the following ping types :
-
An ARP ping, provided the host is on the local subnet
and Nessus is running over Ethernet. -
An ICMP ping.
-
A TCP ping, in which the plugin sends to the remote host
a packet with the flag SYN, and the host will reply with
a RST or a SYN/ACK. -
A UDP ping (e.g., DNS, RPC, and NTP)." n/a
None XXXXXXXX Traceroute Information Makes a traceroute to the remote host. n/a
None XXXXXXXX Microsoft Windows SMB Log In Possible "The remote host is running a Microsoft Windows operating system or
Samba, a CIFS/SMB server for Unix. It was possible to log into it
using one of the following accounts : -
NULL session
-
Guest account
-
Supplied credentials" n/a
None XXXXXXXX Microsoft Windows SMB Shares Enumeration "By connecting to the remote host, Nessus was able to enumerate the
network share names." n/a
None XXXXXXXX Microsoft Windows SMB Shares Access "The remote has one or more Windows shares that can be accessed through
the network with the given credentials.
Depending on the share rights, it may allow an attacker to read /
write confidential data." "To restrict access under Windows, open Explorer, do a right click on
each share, go to the 'sharing' tab, and click on 'permissions'."
None XXXXXXXX Microsoft Windows SMB LsaQueryInformationPolicy Function NULL Session Domain SID Enumeration "By emulating the call to LsaQueryInformationPolicy(), it was possible
to obtain the domain SID (Security Identifier).
The domain SID can then be used to get the list of users of the
domain." n/a
None XXXXXXXX SMB Use Domain SID to Enumerate Users "Using the domain security identifier (SID), Nessus was able to
enumerate the domain users on the remote Windows system." n/a