Logstash grok pattern not _grokparsefailure

Elastic Stack Logstash
1

Hi.

I have a log that I am trying to parse with grok. I tested it on grokconstructor.appspot.com and it groks like i want

image
image1103×409 62.5 KB

but when I use the same snippet in my logstash grok it fails.

log:

I 2024-09-10T14:38:33,454 [vert.x-eventloop-thread-2] logger - Built PermissionsContext: PermissionsContext{principal=JwtPrincipal{jwt='jwt_key'}, roles=[manage-settlement, view-orders, view-trades, manage-risks, manage-orders, manage-summary, manage-optionmodels, view-settlement, view-firms, manage-instruments, view-risks, view-instruments, manage-imports, view-reports, view-optionmodels, view-products], firmType=APPLE, firmCode='FIRM', clearedTradingFirms=[], affiliatedTradingFirms=[], sessionId='6e488596-bb84-4704-87e8-92ad2fec6475', username='service', isSecure=false, validUntil=1725997383}

grok pattern:

Built PermissionsContext: PermissionsContext\{principal=JwtPrincipal\{jwt='%{DATA:jwt}'\}, roles=\[%{GREEDYDATA:roles}\], firmType=%{WORD:firmType}, firmCode='%{WORD:firmCode}', clearedTradingFirms=\[%{GREEDYDATA:clearedTradingFirms}\], affiliatedTradingFirms=\[%{GREEDYDATA:affiliatedTradingFirms}\], sessionId='%{UUID:sessionId}', username='%{WORD:username}', isSecure=%{WORD:isSecure}, validUntil=%{NUMBER:validUntil}\}
2

Hello,

Please share your logstash pipeline and some sample output that shows how the log is being parsed.

3

Running logstash with

input { generator { count => 1 lines => [ "I 2024-09-10T14:38:33,454 [vert.x-eventloop-thread-2] logger - Built PermissionsContext: PermissionsContext{principal=JwtPrincipal{jwt='jwt_key'}, roles=[manage-settlement, view-orders, view-trades, manage-risks, manage-orders, manage-summary, manage-optionmodels, view-settlement, view-firms, manage-instruments, view-risks, view-instruments, manage-imports, view-reports, view-optionmodels, view-products], firmType=APPLE, firmCode='FIRM', clearedTradingFirms=[], affiliatedTradingFirms=[], sessionId='6e488596-bb84-4704-87e8-92ad2fec6475', username='service', isSecure=false, validUntil=1725997383}" ] } }

output { stdout { codec => rubydebug { metadata => false } } }
filter {
    grok {
        match => { "message" => "Built PermissionsContext: PermissionsContext\{principal=JwtPrincipal\{jwt='%{DATA:jwt}'\}, roles=\[%{GREEDYDATA:roles}\], firmType=%{WORD:firmType}, firmCode='%{WORD:firmCode}', clearedTradingFirms=\[%{GREEDYDATA:clearedTradingFirms}\], affiliatedTradingFirms=\[%{GREEDYDATA:affiliatedTradingFirms}\], sessionId='%{UUID:sessionId}', username='%{WORD:username}', isSecure=%{WORD:isSecure}, validUntil=%{NUMBER:validUntil}\}" }
    }
}

works just fine for me. I get

       "jwt" => "jwt_key",
  "firmType" => "APPLE",
"validUntil" => "1725997383",

etc.

4

I was able to solve it by first using a gsub to remove the "'"

    mutate {
      gsub => [ "[message]", "'", ""]
    }
    grok {
      match => {
        'message' => 'Built PermissionsContext: PermissionsContext\{principal=JwtPrincipal\{jwt=%{GREEDYDATA:jwt}\}, roles=\[%{GREEDYDATA:roles}\], firmType=%{WORD:firmType}, firmCode=%{GREEDYDATA:firmCode}, clearedTradingFirms=\[%{GREEDYDATA:clearedTradingFirms}\], affiliatedTradingFirms=\[%{GREEDYDATA:affiliatedTradingFirms}\], sessionId=%{GREEDYDATA:sessionId}, username=%{GREEDYDATA:username}, isSecure=%{WORD:isSecure}, validUntil=%{NUMBER:validUntil}\}'
      }
      tag_on_failure => [ "jwt-grok-failure" ]
      add_tag => [ "grokked" ]
    }
    # Convert validUntil from epoch to UTC
    date {
      match => [ "validUntil", "UNIX" ]
      target => "validUntil"
    }
    if "jwt-grok-failure" not in [tags] {
      mutate {
        remove_field => [ "[dissect][msg]" ]
      }
    }
  }