Configuring Logstash to read Cisco WSA logs

Elastic Stack Logstash
1

Hi there,

I have installed logstash and have some raw logs which have been transferred to my log server from our Cisco WSA (IronPort S170).

I have run some basic tests, e.g the Hello World test which works fine.
When I edit logstash-simple.conf to look at my directory which holds all of my logs, it goes crazy and gives me the following over and over again for each log (there are millions of them):

 "message" => "10.11.22.145 - - [24/Feb/2017:02:07:55 +0000] \"GET https://store-images.s-microsoft.com:443/image/apps.50048.9007199266243744.bb609be5-c8b1-4124-bb0d-5cfd316d4084.01319388-d2c1-4bb3-95fe-11c8e7779920?format=source HTTP/1.1\" 403 0 TCP_DENIED_SSL:NONE 10 DROP_WEBCAT_7-Block_User_Agents-Match_Certain_User_Agents-DefaultGroup-NONE-NONE-NONE <IW_infr,1.5,1,\"-\",-,-,-,-,\"-\",-,-,-,\"-\",-,-,\"-\",\"-\",-,-,IW_infr,-,\"-\",\"-\",\"Unknown\",\"Unknown\",\"-\",\"-\",0.00,0,-,\"-\",\"-\",-,\"-\",-,-,\"-\",\"-\",-,-,\"-\"> - 1487902075.378 NONE \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2486.0 Safari/537.36 Edge/13.10586\" - 0 \"Infrastructure and Content Delivery Networks\"",
      "@version" => "1",
    "@timestamp" => "2017-02-24T16:50:48.599Z",
          "path" => "/data/incomingdata/wsa/accesslogs.@20170224T000001.s",
          "host" => "myserver.local",
          "type" => "apache_access",
          "tags" => [
        [0] "_grokparsefailure"

I totally expect this as I haven't filters but my mind is totally blown. I have no clue where to start, what I am trying to acheive from this or what to expect! All I know is that we need to see our logs in a more readable format.

My conf file is currently in the most basic format:

input {
  file {
    path => "/data/incomingdata/wsa/*"
    start_position => "beginning"
  }
}

filter {
  if [path] =~ "access" {
    mutate { replace => { "type" => "apache_access" } }
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
  }
  stdout { codec => rubydebug }
}

One line out of my logs typically looks like this:

10.11.22.145 "COMPANY\user@NTLM" - [01/Mar/2017:11:03:34 +0000] "TCP_CONNECT 184.50.167.30:443" 200 0 TCP_MISS_SSL:DIRECT 91 DECRYPT_WBRS_7-DefaultGroup-Internal_Users-DefaultGroup-NONE-NONE-DefaultGroup <IW_comp,5.4,1,"-",-,-,-,-,"-",-,-,-,"-",-,-,"-","-",-,-,IW_comp,-,"-","-","Microsoft Dynamics CRM","Enterprise Applications","Encrypted","-",0.00,0,-,"-","-",-,"-",-,-,"-","-",-,-,"-"> - 1488366214.255 NTLMSSP - - 0 "Computers and Internet"

Can anyone help me?

2

You're trying to use the COMBINEDAPACHELOG grok pattern but your log isn't in combined format. It starts off fine but diverges after the timestamp. Maybe someone has written a filter for this kind of log, but I wouldn't count on it. Start with the original definition of COMBINEDAPACHELOG,

but cut it down so it has a chance of matching:

match => { "message" => '%{IPORHOST:clientip} "%{HTTPDUSER:auth}" %{HTTPDUSER:ident} \[%{HTTPDATE:timestamp}\] %{GREEDYDATA:rest}' }

Then start iterating, adding more to your expression each time and trying the expression against real log lines. Look in the Cisco docs to find out what the various fields mean.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.