I'm brand new to this so please bear with me..
I have a log from an application that has inconsistent lines - as in the content of each line is different - so that might be a challenge in itself...but later.
For now I'm using stdin and stout to let me paste text into a command line and have logstash respond with the matches.
An example line of the log is:
2019-01-12 02:59:54.324 Trace [T24ServiceConnector] Sending OFS request [Tx be17fc06-f7c4-414c-b182-e5d41201fdeb]: ENQUIRY.SELECT,,SOMEUSER//AU0010001,RB.CARD.APP.HEARTBEAT,
So far, my conf file looks like this:
input { stdin { } }
filter
{
grok {
patterns_dir => ["C:\Logstash\patterns"]
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{WORD:LogLevel}" }
}
date {
match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss.SSS" ]
timezone=> "Australia/Sydney"
target => "@timestamp" }
}
output
{
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}
This has mostly been through googling and trial and error. It took a while to get the timstamp bit to work.
The output so far is
{
"@timestamp" => 2019-01-11T15:59:54.324Z,
"@version" => "1",
"timestamp" => "2019-01-12 02:59:54.324",
"LogLevel" => "Trace",
"message" => "2019-01-12 02:59:54.324 Trace [T24ServiceConnector] Sending OFS request [Tx be17fc06-f7c4-414c-b182-e5d41201fdeb]: ENQUIRY.SELECT,,SOMEUSER//AU0010001,RB.CARD.APP.HEARTBEAT,\r",
"host" => "SYMV170150"
}
Now I'm trying to grab the next bit [T24ServiceConnector] but cannot work it out.
If I add another 'WORD'
input { stdin { } }
filter
{
grok {
patterns_dir => ["C:\Logstash\patterns"]
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{WORD:LogLevel} %{WORD:LogSource}" }
}date {
match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss.SSS" ]
timezone=> "Australia/Sydney"
target => "@timestamp" }
}output
{
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}
I just get a parse failure
{
"@version" => "1",
"message" => "2019-01-12 02:59:54.324 Trace [T24ServiceConnector] Sending OFS request [Tx be17fc06-f7c4-414c-b182-e5d41201fdeb]: ENQUIRY.SELECT,,SOMEUSER//AU0010001,RB.CARD.APP.HEARTBEAT,\r",
"host" => "SYMV170150",
"@timestamp" => 2019-01-23T22:57:14.953Z,
"tags" => [
[0] "_grokparsefailure"
]
}
I think I need to do some custom filter, Which is why I added the patterns.
Patterns file just contains
LOGSOURCE \W\b\w+\b\W
right now
and replacing WORD with LOGSOURCE for the third parameter gives a parse error also
{
"host" => "SYMV170150",
"@timestamp" => 2019-01-23T23:03:41.845Z,
"message" => "2019-01-12 02:59:54.324 Trace [T24ServiceConnector] Sending OFS request [Tx be17fc06-f7c4-414c-b182-e5d41201fdeb]: ENQUIRY.SELECT,,SOMEUSER//AU0010001,RB.CARD.APP.HEARTBEAT,\r",
"@version" => "1",
"tags" => [
[0] "_grokparsefailure"
]
}
So...I'm looking for pointers on how to grab the bits I want. Ultimately I need the logsource, the entry type, transaction ID and main body as separate entries. I'm hoping that once I get the initial pattern for the square bracket, I can butcher the rest together