Multiline codec applied (without multiline in logs)

Elastic Stack Beats
#1

Hi folks,

Filebeat applies the "multiline coded", even if the log file doesn't contains any multiline logs.

I guess this is related to Drupal log mechanism, as the log file is written every 10sec. with all the logs that occurred during the last 10sec (I assume for less Disk's I/O).

My filebeat is configured as follow:

filebeat:
prospectors:
- input_type: log
paths:
- /home//tomcat/logs/access_2017-03-24.log
fields:
app_type: <myAPP_type>
app_name: <myAPP_name>

output:
logstash:
enabled: true
hosts: ["ELK:5045"]

Logstash is configured with the following pattern:

            match => { 'message' => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:logdate}\] "(?:%{WORD:method} %{URIPATH:URLpath}(?:%{URIPARAM:URLquery})?(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:responseCode} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:responseTime} \[id\: %{WORD:XSolidRequestID}\]'}

And the context.xml is the following:
pattern="%{X-Forwarded-For}i %l %u %t "%r" %s %b "%{Referer}i" "%{User-Agent}i" %D [id: %{X-Request-ID}r]"

Many thanks in advance!
Greg

#2
  • Could you share some example log output from the drupal?
  • Do you have single and multiline mixed in one log file?
  • Could you share your full logstash config? Use 3 ` (ticks) around the code to preserve indentation
#3

Hi @ruflin,

Thanks for your response.

Here some logs output:

10.10.10.50 - - [24/Mar/2017:23:59:56 +0100] "GET /myAPP/myAPPapi/quotes/1102994-SWX-CHF?fields=CASH_LINK,M_NAME,M_TREND,M_CUR:value:id,SC_GROUPED,M_VALOR,M_SYMB,M_MARKET:value:id,M_ISIN,COMPFULLNAME,LVAL,I_NET_VPERPR_V,I_NET_VPERPR_V_PR,EUSIPA:id&autologin=myAPP_0 HTTP/1.0" 200 783 "-" "Drupal (+http://drupal.org/)" 38 [id: 1a5c557a6790c80eaf40dcec8b58e170]
10.10.10.50 - - [24/Mar/2017:23:59:58 +0100] "GET /myAPP/myAPPapi/quotes/833034-MUC-EUR?fields=CASH_LINK,M_NAME,M_TREND,M_CUR:value:id,SC_GROUPED,M_VALOR,M_SYMB,M_MARKET:value:id,M_ISIN,COMPFULLNAME,LVAL,I_NET_VPERPR_V,I_NET_VPERPR_V_PR,EUSIPA:id&autologin=myAPP_0 HTTP/1.0" 200 628 "-" "Drupal (+http://drupal.org/)" 39 [id: b3cb93b8da5136c37a4341303437247a]
10.10.10.50 - - [24/Mar/2017:23:59:58 +0100] "GET /myAPP/ext/fragments/details/fields/833034%2CMUC%2CEUR?style=fragment.detail.equ.aktuelledaten.1&tableId=aktuelledaten-1&decorator=markupOnly HTTP/1.0" 200 1970 "-" "Drupal (+http://drupal.org/)" 18 [id: 97b013d3163575f3f8287a54fd63b611]
10.10.10.50 - - [24/Mar/2017:23:59:58 +0100] "GET /myAPP/ext/fragments/details/fields/833034%2CMUC%2CEUR?style=fragment.detail.equ.aktuelledaten.2&tableId=aktuelledaten-2&decorator=markupOnly HTTP/1.0" 200 2433 "-" "Drupal (+http://drupal.org/)" 12 [id: ebef4d653a12fd3879fdfa62120a2df4]
10.10.10.50 - - [24/Mar/2017:23:59:58 +0100] "GET /myAPP/myAPPapi/quotes/833034-muc-eur?fields=M_MARKET:id&autologin=myAPP_0 HTTP/1.0" 200 332 "-" "Drupal (+http://drupal.org/)" 34 [id: 91f0fe9a72b610344f702c4f82fef8c7]

No, it should only be single line inside

Regarding the logstash config:
01-input:

beats {
port => 5045
ssl => true
ssl_certificate => "logstash-forwarder.crt"
ssl_key => "logstash-forwarder.key"
}

10- filter:

if [fields][app_type] == "myAPP" {
grok {
match => { 'message' => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:logdate}] "(?:%{WORD:method} %{URIPATH:URLpath}(?:%{URIPARAM:URLquery})?(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:responseCode} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:responseTime} [id: %{WORD:XRequestID}]'}
}
mutate {
convert => { "responseTime" => "integer" }
}
date {
match => [ "logdate", "dd/MMM/YYYY:HH:mm:ss Z" ]
target => "logdate"
}

20-Output

} else if [fields][app_name] == "myAPP" {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "log-myAPP-%{+YYYY.MM.dd}"
user =>
password =>
}

#4

having no multiline configured in filebeat, filebeat will not execute any multiline logic at all. Filebeat splits the log file on \r\, \n or \r\n symbol. Have you checked you log-file in an hex-editor containing newline symbols (0x0A, 0x0D) ?

#5

Hi guys,

It was my mistake... I had the wrong port configured on a host.
Thus, it was trying to apply the multiline pattern.

Thanks for the your help

#6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.