Add x-server header to kibana parsing

[sandikata (Росен Александров)]

Hello, i am trying to add http_x_server field/pattern to my installation, but without success i cannot even find any info about custom patterns or so.

There's my current nginx pattern file.

NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:http_host} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response_code} (?:%{NUMBER:bytes}|-) \"%{DATA:referrer}\" \"%{DATA:agent}\" \"%{DATA:request_body}\" (%{IPORHOST:x_forwarded_for})?
NGINXACCESS %{IPORHOST:http_host} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response_code} (?:%{NUMBER:bytes}|-) \"%{DATA:referrer}\" \"%{DATA:agent}\" \"%{DATA:request_body}\" (%{IPORHOST:x_forwarded_for})?

Any suggestions are welcome. Thanks in advance.

[Badger]

And what does your data look like, and how is your grok filter configured?

[sandikata (Росен Александров)]

Hello!
My filter for nginx -> https://termbin.com/vwhi

And sample log.

**********.net 34.250.58.62 [11/Aug/2020:07:03:04 +0000] "GET /api/v1/brokers/accounts/apis/deposits.php?dateFrom=2020-07-14+00%3A00%3A00&dateTo=2020-08-11+23%3A23%3A59&page=1&itemPerPage=1000&offerHash=&apiKey=********-****-****-****-******** HTTP/1.1" 200 258 "-" "Link" "-" "34.250.58.62" "melon" "-"

I need to add apiKey field and x-server field which must display "melon" in this case of "-" if there's no header.

[Badger]

@sandikata

match => { "message" => "%{NGINXACCESS}" }

And what is the definition of the NGINXACCESS pattern?

[sandikata (Росен Александров)]

Take a look at the very first post. In formatted field.

[Badger]

Not sure if I understood the ask, but I suggest you update the grok pattern to be

"NGINXACCESS" => "%{IPORHOST:http_host} %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response_code} (?:%{NUMBER:bytes}|-) \"%{DATA:referrer}\" \"%{DATA:agent}\" \"%{DATA:request_body}\" \"(%{IPORHOST:x_forwarded_for})?\" \"(%{IPORHOST:x-server})?\""

and also add

    grok { match => { "request" => "\?%{GREEDYDATA:querystring}" } }
    kv { source => "querystring" target => "[@metadata][kvData]" field_split => "&" value_split => "=" }
    if [@metadata][kvData][apiKey] {
        mutate { add_field => { "apiKey" => "%{[@metadata][kvData][apiKey]}" } }
    }