How do I match a newline in logstash grok

this is my log sample as below:

[2022-08-07T15:57:54+08:00] "Request-Method-URL: GET XXX Sex - Free Porn Videos on" "Status-Code: 200" "Request-Length: 1" "Request-Time: 1.23" "Upstream-Server:" "Upstream-Status: 200" "Upstream-Response-Length: 1" "Upstream-Response-Time: 1.23" "HTTP-User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36" "HTTP-Referer:" "X-Cache-Status: " "If-Login: 1" "Username: test" "Cookie: abcdefgh" "Request: test" "Country_code: US" "Response: HTTP/1.1 1234
Server: test/1.1
Content-Length: 99
Date: Sun, 07 Aug 2022 07:56:33 GMT"

this my logstash configure as below:

mutate {
	gsub => ["message", "\n", ""]
grok {
	match => { "message" => [
		"(?m)\s*\[%{TIMESTAMP_ISO8601:time_local}\] %{IP:user_ip} (\"Request-Method-URL: (%{WORD:method})? %{URIPROTO:uriproto}://(?:%{URIHOST:urihost})?(?:%{URIPATHPARAM:uripath})?\")? (\"Status-Code: (%{INT:status})?\")? (\"Request-Length: (%{INT:request_length})?\")? (\"Request-Time: (%{BASE10NUM:request_time})?\")? (\"Upstream-Server: (%{IPORHOST:upstream_server})?(:?%{POSINT:upstream_server_port})?\")? (\"Upstream-Status: (%{INT:upstream_status})?\")? (\"Upstream-Response-Length: (%{INT:upstream_response_length})?\")? (\"Upstream-Response-Time: (%{BASE10NUM:upstream_response_time})?\")? (%{QS:http_user_agent})? (\"HTTP-Referer: (%{GREEDYDATA:http_referer})?\")? (\"X-Cache-Status: (%{GREEDYDATA:X_Cache_Status})?\")? (\"If-Login: (%{INT:if_login})?\")? (\"Username: (%{USERNAME:username})?\")? (\"Cookie: (%{GREEDYDATA:cookie})?\")? (\"Request: (%{GREEDYDATA:request})?\")? (\"Country_code: (%{GREEDYDATA:country_code})?\")? (?<response>(.|\r|\n)*)?"

When I try to debug on the, it seems acceptable, the response will show
"Response: HTTP/1.1 1234 \nServer: test/1.1\nContent-Length: 99\nDate: Sun, 07 Aug 2022 07:56:33 GMT\n\n9.9.9.9"

but when I officially use it in ELK, it cannot display the response, the response will be divided into 5 logs as below:


What input configuration are you using?

Hi, this is input configure of logstash.conf

input {
    beats {
        port => 5044
        host => ""
        type => "beats"

You have a multiline log, Logstash is receiving your lines as independent events.

You need to configura multiline in Filebeat, check this documentation.

Hi, this is my filebeat.yml

#===================== Filebeat prospectors =======================
#- paths:
#  - /var/log/nginx/http.acc
#  - /var/log/nginx/http.err
  multiline.type: pattern
  multiline.pattern: '^\['
#  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after

and using filebeat module : /etc/filebeat/modules.d/nginx.yml

# Module: nginx
# Docs:

- module: nginx
  # Access logs
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
      - /var/log/nginx/http.acc


I tested many times, but still can't complete analysis logs.

Can tell me what's wrong with my configure, thanks.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.