Logstash parsing with combinedapachelog pattern for nginx

Elastic Stack Logstash
1

Hello everyone!!

I ran into a problem with parsing my log messages with logstash. I have set ELK stack for geoip monitoring and my logstash config file looks like that:

input {
  beats {
    port => 5044
  }
}


filter {
  grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
  geoip { source => "clientip" }
}


output {
  elasticsearch {
   hosts => ["localhost:9200"]
   index => "geoip"
  }
}

It works good for messages like : 111.22.333.444 - - [15/Oct/2019:07:30:20 +0200] "POST /asifuheahsfeasd/asfef-sasde HTTP/1.1" 200 428 "-" "-"

My goal is to be able to monitor status codes like 200 in this case, IP adresses and "end points" that are here located in last ("-" "-") fields.

I have logs coming from nginx now and configurated like:

log_format extended_access_log
        '$remote_addr - $remote_user [$time_local] '
        '[$scheme://$server_name:$server_port] '
        '"$request" $status $body_bytes_sent '
        '"$http_referer" "$http_user_agent" '
        '(rt=$request_time urt=$upstream_response_time uct=$upstream_connect_time uht=$upstream_header_time)';

So all messages have the following format:

111.22.333.444 - - [08/Jan/2020:11:50:15 +0100] [https://domain.point.name:111] "POST /sadead/asdeadeade/v2/asdeadead/asdead HTTP/1.1" 204 0 "-" "-" (rt=0.111 urt=0.222 uct=0.000 uht=0.444)
  1. How can I parse logs in this format like there is gonna be always IP address at the first place, date, domain , message, status code etc.
  2. I would like to additionaly parse the domain part [https://domain.point.name:111] like protocol:http, name: domain, name2: point, name3: name, port: 111 etc. is it possible to do?

Thank you very much for any help!!!

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.