Conflicting conf files

Hi,

So I've discovered that my latest conf file I've written conflicts with an existing one I designed to take logs from my FW. I'm struggling to understand why having them both together in my confs folder should cause an issue. I understand that Logstash will mash up all the conf files in the designated conf directory and run them as one (or that's what I believe happens), so I guess this is where the issue is arising. My suspicion is that it's to do with using a custom template on my FW conf file. I've shown below only the parts of the 2 conflicting confs in the hope that someone can help me out.

Many thanks.

FW conf...

input {
    udp {
    type => "fw-traffic"
    port => 5513
    }
}

filter {
  if [type] == "fw-traffic" {
    grok {
      patterns_dir => "C:\ELK\logstash\config\patterns"
      match => [ "message", "%{SYSLOGTIMESTAMP:@timestamp} %{HOSTNAME} %{POSINT},%{GREEDYDATA:raw_message}" ]
	}
  }
  
    csv {
      source => "raw_message"
      columns => [ "ReceiveTime","SerialNum","Type","Threat-ContentType","ConfigVersion","GenerateTime","SourceAddress","DestinationAddress","NATSourceIP","NATDestinationIP","Rule","SourceUser","DestinationUser","Application","VirtualSystem","SourceZone","DestinationZone","InboundInterface","OutboundInterface","Log_Forwarding_Profile","TimeLogged","SessionID","RepeatCount","SourcePort","DestinationPort","NATSourcePort","NATDestinationPort","Flags","IPProtocol","Action","Bytes","BytesSent","BytesReceived","Packets","StartTime","ElapsedTimeInSec","Category","Padding","seqno","actionflags","SourceCountry","DestinationCountry","cpadding","pkts_sent","pkts_received","SessionEndReason","DG_Hier_Level_1","DG_Hier_Level_2","DG_Hier_Level_3","DG_Hier_Level_4","vsys_name","DeviceName","ActionSource" ]
    }
    date {
      timezone => "Europe/London"
      match => [ "GenerateTime", "YYYY/MM/dd HH:mm:ss" ]
    }

} #end filter block

output {
  if [type] == "fw-traffic" {

    if "_grokparsefailure" in [tags] {
    # write events that didn't match to a file
    file { "path" => "C:\ELK\logstash\logs\grok-failures.txt" }
	}
  else {
    elasticsearch {
    index => "firewall-traffic-%{+YYYY.MM.dd}"
    hosts => ["172.15.16.2:9200"]
	user => elastic
    password => changeme
    template => "C:\ELK\logstash\config\templates\fw-template.json"
    template_overwrite => false
    }
   }
   }
	}

Anti-Virus conf:

input {
      file {
        type => "sophoslog"
    	path => "C:\ELK\Import_Logs_Folder\DefaultCommonEvents.log"
    	start_position => beginning
    	sincedb_path => "/dev/null"
      }
    }

    filter {
      if [type] == "sophoslog" {
      kv {
      field_split => ";"
      value_split => "="
      trimkey => "\s"
         }
      }

    date {
            match => [ "InsertedAt", "YYYY-MM-dd HH:mm:ss" ]
            target => "@timestamp"
         }
    	 
    mutate {
    	gsub => [ "UserName", "^(.*)[\\\\]", "" ]
    	remove_field => [ "message" ]
    	#strip => [ "UserName" ]
    	gsub => [ "ComputerIPAddress", "\r", "" ]
    }
    }
    output {
    	if [type] == "sophoslog" {
        elasticsearch {
        index => "sophoslog-%{+YYYY.MM.dd}"
        hosts => ["172.15.16.2:9200"]
        user => elastic
        password => changeme
        }
    	stdout { codec => rubydebug }
    }
    }

Running these 2 confs from the same folder forces Logstash to output the following for the AV index:

{
      "path" => "C:\\ELK\\Import_Logs_Folder\\DefaultCommonEvents.log",
"@timestamp" => 2017-06-15T18:28:15.546Z,
  "@version" => "1",
      "host" => "ELK",
      "type" => "sophoslog"
}

Whereas running just the AV conf file will produce the expected results:

{
"Action" => "Read only",
"UserName" => "SYSTEM",
"EventTypeID" => "6",
"ComputerDomain" => "DOMAIN",
"type" => "sophoslog",
"path" => "C:\ELK\Import_Logs_Folder\DefaultCommonEvents.log",
"ActionID" => "5",
"@timestamp" => 2017-06-15T16:44:12.000Z,
"ReportingName" => "BUFFALO HD-PCFU3 USB Device/USBSTOR\DISK&VEN_BUFFALO&PROD_HD-PCFU3&REV_0000\0000031400005F62&0",
"EventTime" => "2017-06-15 17:44:13",
"@version" => "1",
"host" => "ELK",
"ComputerName" => "WORKSTATION2",
"ComputerIPAddress" => "172.16.4.2",
"EventID" => "9750",
"InsertedAt" => "2017-06-15 17:44:12"
}

I don't understand completely what's going on, but you need to be very diligent with your conditionals to make sure filters and outputs apply correctly. Both your filter blocks have conditionals but they don't encompass all filters in that file which I can only assume is incorrect. For example, while the kv filter in your AV file only applies to sophoslog events, the date and mutate filters apply to all events.

Hi Magnus,

Thanks for the pointer. You're right the IF statement closed too soon. That's now been resolved which I was pretty confident would resolve it. It hasn't unfortunately. You said that both filter blocks don't encompass all filters but I can't see where. Using a markup tool, I can trace the opening and closing curly brackets for the filter and they encompass everything directly after the input and close just before the output section, both in the FW conf and the AV conf. Do you see something I don't in the FW conf?!

Here's the previously posted config with the problematic line highlighted:

filter {
  if [type] == "fw-traffic" {
    grok {
      patterns_dir => "C:\ELK\logstash\config\patterns"
      match => [ "message", "%{SYSLOGTIMESTAMP:@timestamp} %{HOSTNAME} %{POSINT},%{GREEDYDATA:raw_message}" ]
	}
  }  # end of if [type] == "fw-traffic"
  
    csv {
      source => "raw_message"
      columns => [ "ReceiveTime","SerialNum","Type","Threat-ContentType","ConfigVersion","GenerateTime","SourceAddress","DestinationAddress","NATSourceIP","NATDestinationIP","Rule","SourceUser","DestinationUser","Application","VirtualSystem","SourceZone","DestinationZone","InboundInterface","OutboundInterface","Log_Forwarding_Profile","TimeLogged","SessionID","RepeatCount","SourcePort","DestinationPort","NATSourcePort","NATDestinationPort","Flags","IPProtocol","Action","Bytes","BytesSent","BytesReceived","Packets","StartTime","ElapsedTimeInSec","Category","Padding","seqno","actionflags","SourceCountry","DestinationCountry","cpadding","pkts_sent","pkts_received","SessionEndReason","DG_Hier_Level_1","DG_Hier_Level_2","DG_Hier_Level_3","DG_Hier_Level_4","vsys_name","DeviceName","ActionSource" ]
    }
    date {
      timezone => "Europe/London"
      match => [ "GenerateTime", "YYYY/MM/dd HH:mm:ss" ]
    }

}

Wow, that was embarrassing......

Thanks. I'm still learning but I didn't think I'd have missed something that obvious!

Thanks for taking the time to help out. :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.