Custom nginx module log format error

Elastic Stack Beats
1

my filebeat nginx log format:

  "\"?%{IP_LIST:nginx.access.remote_ip_list} - %{DATA:nginx.access.user_name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{GREEDYDATA:nginx.access.info}\" %{NUMBER:nginx.access.response_code} %{NUMBER:nginx.access.body_sent.bytes} \"%{DATA:nginx.access.referrer}\" \"%{DATA:nginx.access.agent}\" \"%{DATA:nginx.access.x_forwarded}\" \"%{DATA:nginx.access.request_time}\" \"%{DATA:nginx.access.upstream_addr}\" %{NUMBER:nginx.access.upstream_status} \"%{DATA:nginx.access.upstream_response_time}\" %{NUMBER:nginx.access.upstream_cache_status}"

my nginx log :

219.148.158.41 - - [17/Oct/2018:16:11:05 +0800] "POST /userapi/users/login/signin HTTP/1.1" 400 65 "https://monitoring-beta.cloudwise.com/users/login" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36" "-" "0.007" "192.168.2.10:8002" 400 "0.007" -

But I get a error.message from kibana:

Provided Grok expressions do not match field value: [219.148.158.41 - - [17/Oct/2018:16:11:05 +0800] \"POST /userapi/users/login/signin HTTP/1.1\" 400 65 \"https://monitoring-beta.cloudwise.com/users/login\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36\" \"-\" \"0.007\" \"192.168.2.10:8002\" 400 \"0.007\" -]

So please help me and Thanks.

2

Hi @damonops :slight_smile:

I don't see any matching expression that uses IP_LIST. Maybe you want to use IP instead?

Also, as the message implies, your Grok pattern doesn't match your log line. For example, your pattern looks for an IP and a User name but your log starts with an IP (good), some double dash and a date (no user here)

You can test Grok patterns here: http://grokconstructor.appspot.com/do/match

3

Thank you，sorry,Less content is provided, and more info

Default nginx module log format:

"grok": {
      "field": "message",
      "patterns":[
        "\"?%{IP_LIST:nginx.access.remote_ip_list} - %{DATA:nginx.access.user_name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{GREEDYDATA:nginx.access.info}\" %{NUMBER:nginx.access.response_code} %{NUMBER:nginx.access.body_sent.bytes} \"%{DATA:nginx.access.referrer}\" \"%{DATA:nginx.access.agent}\""
        ],
      "pattern_definitions": {
        "IP_LIST": "%{IP}(\"?,?\\s*%{IP})*"
      },
      "ignore_missing": true
    }
  }

I just add some fileds,like:

"grok": {
      "field": "message",
      "patterns":[
        "\"?%{IP_LIST:nginx.access.remote_ip_list} - %{DATA:nginx.access.user_name} \\[%{HTTPDATE:nginx.access.time}\\] \"%{GREEDYDATA:nginx.access.info}\" %{NUMBER:nginx.access.response_code} %{NUMBER:nginx.access.body_sent.bytes} \"%{DATA:nginx.access.referrer}\" \"%{DATA:nginx.access.agent}\" \"%{DATA:nginx.access.x_forwarded}\" \"%{DATA:nginx.access.request_time}\" \"%{DATA:nginx.access.upstream_addr}\" %{NUMBER:nginx.access.upstream_status} \"%{DATA:nginx.access.upstream_response_time}\" %{NUMBER:nginx.access.upstream_cache_status}"
        ],
      "pattern_definitions": {
        "IP_LIST": "%{IP}(\"?,?\\s*%{IP})*"
      },
      "ignore_missing": true
    }
  }

See other information in default file:

/usr/share/filebeat/module/nginx/access/ingest/default.json

Oh, my filebeat version is 6.3.0,thanks

4

I hope there is a example in the documents about customing log module format.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.