Access custom fields on nginx from logstash


(Jonghun Park) #1

I've wrote a nginx log_format as below.

log_format main '$http_host $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" "$http_user_browser_language" $sent_http_sv_il $sent_http_sv_cid $sent_http_sv_uid "User-Time-Zone=$http_User_Time_Zone"';

And I want to filter message for $sent_http_sv_il and $sent_http_sv_cid, $sent_http_sv_uid from Logstash for getting informations more details.

I tried to grok filter as below.

add_field => { "sv_cid" => "%{sent_http_sv_cid}" "sv_uid" => "%{sent_http_sv_uid}"}

the result of them as below

"sv_uid": "%{sent_http_sv_uid}"

what's the missing?
anyone who answer me the way?


(Magnus Bäck) #2

You need to adjust the grok expression you use to also extract the new fields. If you show us your current grok expression we can give more specific advice.


(Jonghun Park) #3

I'm very appreciate for your reply.

it is our current grok expression shown below.

grok {
match => { "message" => ["%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} [%{HTTPDATE:[nginx][access][time]}] "%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} "%{DATA:[nginx][access][agent]}" "%{IPORHOST:[nginx][access][remote_ip_list]}""] }
remove_field => "message"
add_field => { "sv_cid" => "%{NUMBER:[nginx][access][sent_http_sv_uid]}" "sv_uid" => "%{sent_http_sv_uid}" }
}


(Magnus Bäck) #4

Okay, so your current grok expression only covers up to and including "$http_x_forwarded_for". After that you have "$http_user_browser_language" $sent_http_sv_il $sent_http_sv_cid $sent_http_sv_uid so you need to append something like

%{QS} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE:sv_cid} %{NOTSPACE:sv_uid}

to capture the additional fields you're interested in.


(Jonghun Park) #5

awesome!

it works as intended.
And I'm very appreciate for your help!!

best regards.


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.