Please help with pattern

Elastic Stack Logstash
1

My nginx access log format:

log_format  main  '[$time_local] $remote_addr [$geoip_city_country_code:$geoip_region_name:$geoip_city] - "$status" "$http_host" '
            '"$request"  $body_bytes_sent [$request_time s]  "$http_referer" "$http_user_agent"';

log_format upstream '[$time_local] $remote_addr [$geoip_city_country_code:$geoip_region_name:$geoip_city] -> $upstream_addr "$upstream_status" '
            '[$upstream_connect_time sec] [$upstream_response_time sec] "$http_host" "$request"  "$http_referer" "$http_user_agent"';

Help me plz with logstash config

Thank you!

2

Hello vitich,

You should have a raw log generated by nginx with this custom format, it's really more easy to build grok regex.

Then with a single log, build your regex online ( https://grokdebug.herokuapp.com/) or through Kibana / Dev tools / Grok Debugger (Kibana 5.5 or earlier). Use grok patterns is better (smaller and efficient regex).

Finally, use grok plugin in Logstash to match your message field with your regex.

3

[%{HTTPDATE:timestamp}] %{IPORHOST:clientip} [..:-:-] (?:%{USER:nginx_user_ident}|-) "%{NUMBER:nginx_response}" "%{IPORHOST:http_host}" "(?:%{WORD:nginx_http_request} %{URIPATHPARAM:nginx_request_desc}(?: HTTP/%{NUMBER:nginx_http_version})?|-)" %{NUMBER:bytes} [%{NUMBER:response} s] (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent}

Almost done but I don`t know what to do with geoip part [..:-:-]

4

Re,

If the GeoIP resolution (IP address into localisation) is done by Nginx, you can parse directly the localisation into ES document fields. it seem the case.
Otherwise, use the Logstasth GeoIP plugin after grok.

5

The problem is solved. Thank you.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.