rosselg
(Greg)
March 24, 2017, 9:35am
1
Hi folks,
Filebeat applies the "multiline coded", even if the log file doesn't contains any multiline logs.
I guess this is related to Drupal log mechanism, as the log file is written every 10sec. with all the logs that occurred during the last 10sec (I assume for less Disk's I/O).
My filebeat is configured as follow:
filebeat:
prospectors:
- input_type: log
paths:
- /home//tomcat/logs/access_2017-03-24.log
fields:
app_type: <myAPP_type>
app_name: <myAPP_name>
output:
logstash:
enabled: true
hosts: ["ELK:5045"]
Logstash is configured with the following pattern:
match => { 'message' => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:logdate}\] "(?:%{WORD:method} %{URIPATH:URLpath}(?:%{URIPARAM:URLquery})?(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:responseCode} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:responseTime} \[id\: %{WORD:XSolidRequestID}\]'}
And the context.xml is the following:
pattern="%{X-Forwarded-For}i %l %u %t "%r" %s %b "%{Referer}i" "%{User-Agent}i" %D [id: %{X-Request-ID}r]"
Many thanks in advance!
Greg
rosselg
(Greg)
March 27, 2017, 9:54am
3
Hi @ruflin ,
Thanks for your response.
Here some logs output:
10.10.10.50 - - [24/Mar/2017:23:59:56 +0100] "GET /myAPP/myAPPapi/quotes/1102994-SWX-CHF?fields=CASH_LINK,M_NAME,M_TREND,M_CUR:value:id,SC_GROUPED,M_VALOR,M_SYMB,M_MARKET:value:id,M_ISIN,COMPFULLNAME,LVAL,I_NET_VPERPR_V,I_NET_VPERPR_V_PR,EUSIPA:id&autologin=myAPP_0 HTTP/1.0" 200 783 "-" "Drupal (+http://drupal.org/)" 38 [id: 1a5c557a6790c80eaf40dcec8b58e170]
10.10.10.50 - - [24/Mar/2017:23:59:58 +0100] "GET /myAPP/myAPPapi/quotes/833034-MUC-EUR?fields=CASH_LINK,M_NAME,M_TREND,M_CUR:value:id,SC_GROUPED,M_VALOR,M_SYMB,M_MARKET:value:id,M_ISIN,COMPFULLNAME,LVAL,I_NET_VPERPR_V,I_NET_VPERPR_V_PR,EUSIPA:id&autologin=myAPP_0 HTTP/1.0" 200 628 "-" "Drupal (+http://drupal.org/)" 39 [id: b3cb93b8da5136c37a4341303437247a]
10.10.10.50 - - [24/Mar/2017:23:59:58 +0100] "GET /myAPP/ext/fragments/details/fields/833034%2CMUC%2CEUR?style=fragment.detail.equ.aktuelledaten.1&tableId=aktuelledaten-1&decorator=markupOnly HTTP/1.0" 200 1970 "-" "Drupal (+http://drupal.org/)" 18 [id: 97b013d3163575f3f8287a54fd63b611]
10.10.10.50 - - [24/Mar/2017:23:59:58 +0100] "GET /myAPP/ext/fragments/details/fields/833034%2CMUC%2CEUR?style=fragment.detail.equ.aktuelledaten.2&tableId=aktuelledaten-2&decorator=markupOnly HTTP/1.0" 200 2433 "-" "Drupal (+http://drupal.org/)" 12 [id: ebef4d653a12fd3879fdfa62120a2df4]
10.10.10.50 - - [24/Mar/2017:23:59:58 +0100] "GET /myAPP/myAPPapi/quotes/833034-muc-eur?fields=M_MARKET:id&autologin=myAPP_0 HTTP/1.0" 200 332 "-" "Drupal (+http://drupal.org/)" 34 [id: 91f0fe9a72b610344f702c4f82fef8c7]
No, it should only be single line inside
Regarding the logstash config :
01-input:
beats {
port => 5045
ssl => true
ssl_certificate => "logstash-forwarder.crt"
ssl_key => "logstash-forwarder.key"
}
10- filter:
if [fields][app_type] == "myAPP" {
grok {
match => { 'message' => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} [%{HTTPDATE:logdate}] "(?:%{WORD:method} %{URIPATH:URLpath}(?:%{URIPARAM:URLquery})?(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:responseCode} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:responseTime} [id: %{WORD:XRequestID}]'}
}
mutate {
convert => { "responseTime" => "integer" }
}
date {
match => [ "logdate", "dd/MMM/YYYY:HH:mm:ss Z" ]
target => "logdate"
}
20-Output
} else if [fields][app_name] == "myAPP" {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "log-myAPP-%{+YYYY.MM.dd}"
user =>
password =>
}
steffens
(Steffen Siering)
March 28, 2017, 8:14am
4
having no multiline configured in filebeat, filebeat will not execute any multiline logic at all. Filebeat splits the log file on \r\
, \n
or \r\n
symbol. Have you checked you log-file in an hex-editor containing newline symbols (0x0A, 0x0D) ?
rosselg
(Greg)
March 28, 2017, 10:34am
5
Hi guys,
It was my mistake... I had the wrong port configured on a host.
Thus, it was trying to apply the multiline pattern.
Thanks for the your help
system
(system)
Closed
April 25, 2017, 10:34am
6
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.