How do I match a newline in logstash grok

Elastic Stack Logstash
1

this is my log sample as below:

[2022-08-07T15:57:54+08:00] 9.9.9.9 "Request-Method-URL: GET XXX Sex - Free Porn Videos on XXX.com" "Status-Code: 200" "Request-Length: 1" "Request-Time: 1.23" "Upstream-Server: 1.1.1.1:10000" "Upstream-Status: 200" "Upstream-Response-Length: 1" "Upstream-Response-Time: 1.23" "HTTP-User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36" "HTTP-Referer: https://www.google.com" "X-Cache-Status: " "If-Login: 1" "Username: test" "Cookie: abcdefgh" "Request: test" "Country_code: US" "Response: HTTP/1.1 1234
Server: test/1.1
Content-Length: 99
Date: Sun, 07 Aug 2022 07:56:33 GMT

9.9.9.9"

this my logstash configure as below:

mutate {
	gsub => ["message", "\n", ""]
}
grok {
	match => { "message" => [
		"(?m)\s*\[%{TIMESTAMP_ISO8601:time_local}\] %{IP:user_ip} (\"Request-Method-URL: (%{WORD:method})? %{URIPROTO:uriproto}://(?:%{URIHOST:urihost})?(?:%{URIPATHPARAM:uripath})?\")? (\"Status-Code: (%{INT:status})?\")? (\"Request-Length: (%{INT:request_length})?\")? (\"Request-Time: (%{BASE10NUM:request_time})?\")? (\"Upstream-Server: (%{IPORHOST:upstream_server})?(:?%{POSINT:upstream_server_port})?\")? (\"Upstream-Status: (%{INT:upstream_status})?\")? (\"Upstream-Response-Length: (%{INT:upstream_response_length})?\")? (\"Upstream-Response-Time: (%{BASE10NUM:upstream_response_time})?\")? (%{QS:http_user_agent})? (\"HTTP-Referer: (%{GREEDYDATA:http_referer})?\")? (\"X-Cache-Status: (%{GREEDYDATA:X_Cache_Status})?\")? (\"If-Login: (%{INT:if_login})?\")? (\"Username: (%{USERNAME:username})?\")? (\"Cookie: (%{GREEDYDATA:cookie})?\")? (\"Request: (%{GREEDYDATA:request})?\")? (\"Country_code: (%{GREEDYDATA:country_code})?\")? (?<response>(.|\r|\n)*)?"
}

When I try to debug on the https://grokdebug.herokuapp.com, it seems acceptable, the response will show
"Response: HTTP/1.1 1234 \nServer: test/1.1\nContent-Length: 99\nDate: Sun, 07 Aug 2022 07:56:33 GMT\n\n9.9.9.9"

but when I officially use it in ELK, it cannot display the response, the response will be divided into 5 logs as below:

image
image1562×241 16.9 KB

Thanks.

2

What input configuration are you using?

3

Hi, this is input configure of logstash.conf

input {
    beats {
        port => 5044
        host => "0.0.0.0"
        type => "beats"
    }
}
4

You have a multiline log, Logstash is receiving your lines as independent events.

You need to configura multiline in Filebeat, check this documentation.

5

Hi, this is my filebeat.yml

#===================== Filebeat prospectors =======================
filebeat.inputs:
#- paths:
#  - /var/log/nginx/http.acc
#  - /var/log/nginx/http.err
  multiline.type: pattern
  multiline.pattern: '^\['
#  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after

and using filebeat module : /etc/filebeat/modules.d/nginx.yml

# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-module-nginx.html

- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths:
      - /var/log/nginx/http.acc
6

Hi,

I tested many times, but still can't complete analysis logs.

Can tell me what's wrong with my configure, thanks.