Squid logs (access logs)

Elastic Stack Logstash
1

I am trying to create index for a squid log....although the filebeat creates the pipeline for logstash but due to some error in my grok I am not able to segregate the logs..P.S help me..
sample of my log file:

"
551728235.295 866048 10.130.2.58 TCP_TUNNEL/200 8429 CONNECT play.google.com:443 artidesh HIER_DIRECT/172.217.160.206 -
1551728237.817 61121 10.27.2.174 TCP_MISS/200 319 GET http://su.ff.avast.com/R/A28KIGJmYzZlYWM5OGFkNDQzYWI5OWU5MmQ5ZThiY2RmMWVkEgQGBwIZGHgiAf8qBwgEEOS292syCggAELC492sYgAI465eMkAFCINs5qHO9YDIoqtdhpEPo6hYMAcz6Dpa_oZsRDcsK0yfhSICDmBA= sumitk HIER_DIRECT/77.234.45.65 application/octet-stream
1551728238.234 109983 10.26.2.44 TCP_TUNNEL/200 587 CONNECT c.go-mpulse.net:443 harish HIER_DIRECT/104.81.21.87 -
1551728238.997 60974 10.126.2.11 TCP_MISS/200 319 GET http://su.ff.avast.com/R/A3cKIGRkMDc1NTJkYTEwZDRlMGNiZjMwMWJkZDQxY2I2YzUxEgQCAwMZGHgiAf4qBwgEEK-492sqBwgDEKDT8GYyCggEEK-492sYgAo4spKckAFCIAWim9dpoLM1oNFpiMFNau5nnsWSKWVZZP3OPYtUxXoRSICDKA== mukund HIER_DIRECT/77.234.41.236 application/octet-stream"

my grok looks like:

input {
beats {
port => "5044"
}
}
filter {
grok {
match => { "message" => "%{INT:timestamp}.%{INT}\s*%{NUMBER:request_msec:float} %{IPORHOST:src_ip} %{WORD:cache_result}/%{NUMBER:response_status:int} %{NUMBER:response_size:int} %{WORD:http_method} (%{URIPROTO:http_proto}://)?%{IPORHOST:dst_host}(?::%{POSINT:port})?(?:%{DATA:uri_param})? %{USERNAME:cache_user} %{WORD:request_route}/(%{IPORHOST:forwarded_to}|-) %{GREEDYDATA:content_type}"}
}
geoip {
source => "dst_ip"
}

}

output {

elasticsearch {
hosts => ["10.11.109.7:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY-MM}"
document_type => "%{[@metadata][type]}"
}

}

2

Your pattern works for me...

   "request_msec" => 61121.0,
     "cache_user" => "sumitk",
         "src_ip" => "10.27.2.174",
    "http_method" => "GET",
   "forwarded_to" => "77.234.45.65",
   "content_type" => "application/octet-stream",
      "timestamp" => "1551728237",
"response_status" => 200,
  "response_size" => 319,
       "dst_host" => "su.ff.avast.com",
   "cache_result" => "TCP_MISS",
  "request_route" => "HIER_DIRECT",
        "message" => "1551728237.817 61121 10.27.2.174 TCP_MISS/200 319 GET http://su.ff.avast.com/R/A28KIGJmYzZlYWM5OGFkNDQzYWI5OWU5MmQ5ZThiY2RmMWVkEgQGBwIZGHgiAf8qBwgEEOS292syCggAELC492sYgAI465eMkAFCINs5qHO9YDIoqtdhpEPo6hYMAcz6Dpa_oZsRDcsK0yfhSICDmBA= sumitk HIER_DIRECT/77.234.45.65 application/octet-stream",
     "http_proto" => "http",
      "uri_param" => "/R/A28KIGJmYzZlYWM5OGFkNDQzYWI5OWU5MmQ5ZThiY2RmMWVkEgQGBwIZGHgiAf8qBwgEEOS292syCggAELC492sYgAI465eMkAFCINs5qHO9YDIoqtdhpEPo6hYMAcz6Dpa_oZsRDcsK0yfhSICDmBA="
3

But while creating Index in elasticsearch it ain't segregating.
My elasticsearch output is:

|t message| |1551759655.972 103 10.31.2.143 TCP_MISS/206 66117 GET http://2.au.download.windowsupdate.com/c/msdownload/update/software/crup/2017/08/ppaddin-x-none_dafe4123f452d9c1a822e3844dd317e342915b89.cab - HIER_DIRECT/104.118.7.49 application/vnd.ms-cab-compressed|
I am really stuck at this point

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.