How to extract several words as a phrase?

I'm trying to learn how to use grok. I've managed to make some progress, but I am stuck again :frowning:

Example of my log:

2019-01-12 02:59:54.324 Trace    [T24ServiceConnector] Sending OFS request [Tx be17fc06-f7c4-414c-b182-e5d41201fdeb]: ENQUIRY.SELECT,,SOMESUSER//AU0010001,RB.CARD.APP.HEARTBEAT,
2019-01-12 02:59:54.433 Error    [T24ServiceConnector] Error while processing request [Tx cfbd08c6-9fea-4d88-b15b-6d0637418452]: Unable to connect to the configured jagent instances due to unexpected error while processing the request.
2019-01-12 02:59:54.433 Trace    [T24ServiceConnector] Response [Tx cfbd08c6-9fea-4d88-b15b-6d0637418452]: Error while processing request [Tx cfbd08c6-9fea-4d88-b15b-6d0637418452]: Unable to connect to the configured jagent instances due to unexpected error while processing the request.
2019-01-12 02:59:54.433 Trace    [T24ServiceConnector] Received response [Tx cfbd08c6-9fea-4d88-b15b-6d0637418452] from T24ServiceConnector.2019-01-12 02:59:54.324 Trace    [T24ServiceConnector] Sending OFS request [Tx be17fc06-f7c4-414c-b182-e5d41201fdeb]: ENQUIRY.SELECT,,SOMESUSER//AU0010001,RB.CARD.APP.HEARTBEAT,
2019-01-12 02:59:54.433 Error    [T24ServiceConnector] Error while processing request [Tx cfbd08c6-9fea-4d88-b15b-6d0637418452]: Unable to connect to the configured jagent instances due to unexpected error while processing the request.
2019-01-12 02:59:54.433 Trace    [T24ServiceConnector] Response [Tx cfbd08c6-9fea-4d88-b15b-6d0637418452]: Error while processing request [Tx cfbd08c6-9fea-4d88-b15b-6d0637418452]: Unable to connect to the configured jagent instances due to unexpected error while processing the request.
2019-01-12 02:59:54.433 Trace    [T24ServiceConnector] Received response [Tx cfbd08c6-9fea-4d88-b15b-6d0637418452] from T24ServiceConnector.

After the [T24ServiceConnector] and before the Transaction ID, there is between 1 and 4 words.
How do I get that to match?

My code so far:

input { stdin { } }
filter
{
    grok { 
    patterns_dir => ["C:\Logstash\patterns"]
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{WORD:LogLevel} \s*%{BRACKETEDWORD:LogSource} %{PHRASE:Description}" }
  } 
  date {
    match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss.SSS" ]
	timezone=> "Australia/Sydney"
    target => "@timestamp" } 
}
output
{
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}

With a pattern file of;

BRACKETEDWORD \[%{WORD}\]
PHRASE (%{WORD}{1,4})

The phrase isn't working. it only picks up the first word. Clearly I'm misunderstanding how it should work

  • in my other question is was suggested to use pattern definition. I did try, but I'm not getting the syntax correct. I also failed to get file input to work so stuck with stdin for now.

I discovered DATA

input { stdin { } }
filter
{
    grok { 
    patterns_dir => ["C:\Logstash\patterns"]
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{WORD:LogLevel} \s*%{BRACKETEDWORD:LogSource} %{DATA:Description} %{TRANSACTIONID:TxID}.? %{GREEDYDATA:BodyDetail}" }
  } 
  date {
    match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss.SSS" ]
	timezone=> "Australia/Sydney"
    target => "@timestamp" } 
}

output
{
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}

and

BRACKETEDWORD \[%{WORD}\]
TRANSACTIONID (\[Tx [a-zA-Z0-9]{8}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{12}\])

gives me a result

 {
       "LogLevel" => "Trace",
        "message" => "2019-01-12 02:59:54.324 Trace    [T24ServiceConnector] Sending OFS request [Tx be17fc06-f7c4-414c-b182-e5d41201fdeb]: ENQUIRY.SELECT,,SOMESUSER//AU0010001,RB.CARD.APP.HEARTBEAT,\r",
     "BodyDetail" => "ENQUIRY.SELECT,,SOMESUSER//AU0010001,RB.CARD.APP.HEARTBEAT,\r",
           "host" => "SYMV170150",
    "Description" => "Sending OFS request",
           "TxID" => "be17fc06-f7c4-414c-b182-e5d41201fdeb",
       "@version" => "1",
     "@timestamp" => 2019-01-11T15:59:54.324Z,
      "LogSource" => "T24ServiceConnector",
      "timestamp" => "2019-01-12 02:59:54.324"
}
1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.