NGINX Log mapping

Elastic Stack Logstash
1

I setup ELK server (kibana-4.6.1,logstash-2.4,elasticsearch-2.4) I have setup one index pattern (logstash-*) and using one time-field name (@timestamp)

NGUSERNAME [a-zA-Z.@-+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} [%{HTTPDATE:timestamp}] %{HOSTNAME:hostname} "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent}

logstash.conf

My problem is that I cannot use fields (clientip and other), I not see in Kibana this field. Problem in configuration files or kibana ?
Thank you!

2

When run Logstash with the -f flag /opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf, I see in kibana new field (clientip. geoip) . When disable "ctrl-C" and "ctrl-D", I not see in kibana this field. Logstash is running in system

3

Pasted image755×628 6.76 KB

This tag indicates that you have an error in grok parse.

This it´s wrong.

NGUSERNAME [a-zA-Z.@-+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} [%{HTTPDATE:timestamp}] %{HOSTNAME:hostname} "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent}

Your log uses another standard.

4

Unless you show us what kind of message is failing to parse we can't help.

5

Thank you!