I'm having trouble getting the correct output in Kabana from my log files ingested by filebeats. I'm pretty sure the issue is with filebeats though so I'm posting it here. My log files have the following format:
[datetime]\tsource\tseverity\tmessage
The message field is multi-line
filebeat.yml:
paths:
- C:\fuzeapplogs\*
multiline.pattern: ^\[
multiline.negate: true
multiline.match: after
logstash.conf:
input {
beats {
port => 5044
}
}
filter{
csv{
#separator uses an ACTUAL tab, not \t
separator => " "
columns => ["fuzedatetimestamp","fuzesource","fuzeseverity","fuzemessage"]
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "fuzelogs"
}
}
Yet when I look at the data in Kibana discovery tab, the fuzemessage field just has the first line.
For instance, here is a line from my log file:
[5/19/2020 3:39:42 PM] Web error Server: MSI
Virtual Directiory: /qa75
Url: https://localhost/qa75/common/kbsearch.aspx
Raw Url: /qa75/int/kbsearch.aspx
DateTime: 5/19/2020 3:39:42 PM
IP Address: ::1
User Browser: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36
Error Description: Attempted to divide by zero.
Error Type: System.DivideByZeroException
Error Source: App_Web_kbsearch.ascx.8c4a966a.inf_ndgv
QueryString:
Referrer: /qa75/csr/makekb.asp
Target Site: Void CreateDSN()
Stack Trace: at FuzeWeb.ctrls_KBSearch.CreateDSN() in c:\Users\charl\source\repos\FuzeApp\Fuze80Sln\qa75\ctrls\KBSearch.ascx.cs:line 1236
at FuzeWeb.ctrls_KBSearch.Page_Load(Object sender, EventArgs e) in c:\Users\charl\source\repos\FuzeApp\Fuze80Sln\qa75\ctrls\KBSearch.ascx.cs:line 178
at System.Web.UI.Control.OnLoad(EventArgs e)
at System.Web.UI.Control.LoadRecursive()
at System.Web.UI.Control.LoadRecursive()
at System.Web.UI.Control.LoadRecursive()
In Kibana, the fuzedatetimestamp field's value = [5/19/2020 5:08:49 PM], the fuzesource field's value = "Web", the fuzeseverity field's value = "error", but the fuzemessage field value = "Server: MSI" , instead of "Server: MSI
Virtual Directiory: /qa75
Url: https://localhost/qa75/common/kbsearch.aspx
Raw Url: /qa75/int/kbsearch.aspx
DateTime: 5/19/2020 3:39:42 PM
IP Address: ::1
User Browser: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36
Error Description: Attempted to divide by zero.
Error Type: System.DivideByZeroException
Error Source: App_Web_kbsearch.ascx.8c4a966a.inf_ndgv
QueryString:
Referrer: /qa75/csr/makekb.asp
Target Site: Void CreateDSN()
Stack Trace: at FuzeWeb.ctrls_KBSearch.CreateDSN() in c:\Users\charl\source\repos\FuzeApp\Fuze80Sln\qa75\ctrls\KBSearch.ascx.cs:line 1236
at FuzeWeb.ctrls_KBSearch.Page_Load(Object sender, EventArgs e) in c:\Users\charl\source\repos\FuzeApp\Fuze80Sln\qa75\ctrls\KBSearch.ascx.cs:line 178
at System.Web.UI.Control.OnLoad(EventArgs e)
at System.Web.UI.Control.LoadRecursive()
at System.Web.UI.Control.LoadRecursive()
at System.Web.UI.Control.LoadRecursive()"
What am I doing wrong??