Hi,
So I've discovered that my latest conf file I've written conflicts with an existing one I designed to take logs from my FW. I'm struggling to understand why having them both together in my confs folder should cause an issue. I understand that Logstash will mash up all the conf files in the designated conf directory and run them as one (or that's what I believe happens), so I guess this is where the issue is arising. My suspicion is that it's to do with using a custom template on my FW conf file. I've shown below only the parts of the 2 conflicting confs in the hope that someone can help me out.
Many thanks.
FW conf...
input {
udp {
type => "fw-traffic"
port => 5513
}
}
filter {
if [type] == "fw-traffic" {
grok {
patterns_dir => "C:\ELK\logstash\config\patterns"
match => [ "message", "%{SYSLOGTIMESTAMP:@timestamp} %{HOSTNAME} %{POSINT},%{GREEDYDATA:raw_message}" ]
}
}
csv {
source => "raw_message"
columns => [ "ReceiveTime","SerialNum","Type","Threat-ContentType","ConfigVersion","GenerateTime","SourceAddress","DestinationAddress","NATSourceIP","NATDestinationIP","Rule","SourceUser","DestinationUser","Application","VirtualSystem","SourceZone","DestinationZone","InboundInterface","OutboundInterface","Log_Forwarding_Profile","TimeLogged","SessionID","RepeatCount","SourcePort","DestinationPort","NATSourcePort","NATDestinationPort","Flags","IPProtocol","Action","Bytes","BytesSent","BytesReceived","Packets","StartTime","ElapsedTimeInSec","Category","Padding","seqno","actionflags","SourceCountry","DestinationCountry","cpadding","pkts_sent","pkts_received","SessionEndReason","DG_Hier_Level_1","DG_Hier_Level_2","DG_Hier_Level_3","DG_Hier_Level_4","vsys_name","DeviceName","ActionSource" ]
}
date {
timezone => "Europe/London"
match => [ "GenerateTime", "YYYY/MM/dd HH:mm:ss" ]
}
} #end filter block
output {
if [type] == "fw-traffic" {
if "_grokparsefailure" in [tags] {
# write events that didn't match to a file
file { "path" => "C:\ELK\logstash\logs\grok-failures.txt" }
}
else {
elasticsearch {
index => "firewall-traffic-%{+YYYY.MM.dd}"
hosts => ["172.15.16.2:9200"]
user => elastic
password => changeme
template => "C:\ELK\logstash\config\templates\fw-template.json"
template_overwrite => false
}
}
}
}
Anti-Virus conf:
input {
file {
type => "sophoslog"
path => "C:\ELK\Import_Logs_Folder\DefaultCommonEvents.log"
start_position => beginning
sincedb_path => "/dev/null"
}
}
filter {
if [type] == "sophoslog" {
kv {
field_split => ";"
value_split => "="
trimkey => "\s"
}
}
date {
match => [ "InsertedAt", "YYYY-MM-dd HH:mm:ss" ]
target => "@timestamp"
}
mutate {
gsub => [ "UserName", "^(.*)[\\\\]", "" ]
remove_field => [ "message" ]
#strip => [ "UserName" ]
gsub => [ "ComputerIPAddress", "\r", "" ]
}
}
output {
if [type] == "sophoslog" {
elasticsearch {
index => "sophoslog-%{+YYYY.MM.dd}"
hosts => ["172.15.16.2:9200"]
user => elastic
password => changeme
}
stdout { codec => rubydebug }
}
}
Running these 2 confs from the same folder forces Logstash to output the following for the AV index:
{
"path" => "C:\\ELK\\Import_Logs_Folder\\DefaultCommonEvents.log",
"@timestamp" => 2017-06-15T18:28:15.546Z,
"@version" => "1",
"host" => "ELK",
"type" => "sophoslog"
}
Whereas running just the AV conf file will produce the expected results:
{
"Action" => "Read only",
"UserName" => "SYSTEM",
"EventTypeID" => "6",
"ComputerDomain" => "DOMAIN",
"type" => "sophoslog",
"path" => "C:\ELK\Import_Logs_Folder\DefaultCommonEvents.log",
"ActionID" => "5",
"@timestamp" => 2017-06-15T16:44:12.000Z,
"ReportingName" => "BUFFALO HD-PCFU3 USB Device/USBSTOR\DISK&VEN_BUFFALO&PROD_HD-PCFU3&REV_0000\0000031400005F62&0",
"EventTime" => "2017-06-15 17:44:13",
"@version" => "1",
"host" => "ELK",
"ComputerName" => "WORKSTATION2",
"ComputerIPAddress" => "172.16.4.2",
"EventID" => "9750",
"InsertedAt" => "2017-06-15 17:44:12"
}
