Sensitive information parsing

Elastic Stack Logstash
1

Hello All,

I am trying to mask a sensitive information from a log message.

I have given the sample log file and the config file as below. i was trying to do this using gsub with no success. Please review and advise at the earliest.

Sample Log Message :

2018-01-10T16:58:16.925-0700 |[WebContainer : 0]| TRACE | sent | Sent response [<soapenv:Envelope xmlns:soapenv="http://xyz.mlsop.com//envelope/" FirstName>MOHAN1949-12-01

Output from Logstash :

      "Type" => "TRACE",
      "Task" => "sent",
"@timestamp" => 2018-02-20T16:21:35.948Z,
  "Messsage" => " CO Region Expiry days not added to the expiration Date",
  "@version" => "1",
      "host" => "oc1008401175.ibm.com",
   "message" => "2018-01-10T16:58:16.925-0700 |[WebContainer : 0]| TRACE | sent | Sent response [<soapenv:Envelope xmlns:soapenv="http://xyz.mlsop.com//envelope/" FirstName><MiddleName></MiddleName><LastName>MOHAN</LastName><DateOfBirth>1949-12-01</DateOfBirth>",
      "Date" => "2018-01-10T16:58:16.925-0700",
 "MsgSource" => "[WebContainer : 6]",

Config file:

input {
file {
path => "/home/mohank44/Data/6LogAnalysis/PSI/PSItest5.log"
#type => "LOG"
start_position => "beginning"
codec => multiline
{
pattern => "^\A%{TIMESTAMP_ISO8601}%{SPACE}|%{SYSLOG5424SD}|%{SPACE}%{WORD}"
negate => true
what => previous
} }
}
filter{
mutate
{
gsub => ["message", "\n", ""]
gsub => ["message", "\r", ""]
gsub => ["message", "\t", ""]
gsub => ["message", "(%[a-zA-z0-9-]+)", "yyyy-mm-dd"]
}
grok {
match => [ "message", "\A%{TIMESTAMP_ISO8601:Date}%{SPACE}|%{SYSLOG5424SD:MsgSource}|%{SPACE}%{WORD:Type}%{SPACE}|%{SPACE}%{WORD:Task}%{SPACE}|%{GREEDYDATA:Messsage}" ]
overwrite => [ "message" ]
}
date {
match => [ "Date","yyyy-MM-dd'T'HH:mm:ss.SSS-ZZZZ"]
target => "Date"
}

}
output
{
stdout {codec => rubydebug}
#stdout{}
elasticsearch
{
hosts => "localhost"
index => "log-psi-index3"
}
}
Expected Output :

"Message" => "2018-01-10T16:58:16.925-0700 |[WebContainer : 0]| TRACE | sent | Sent response [<soapenv:Envelope xmlns:soapenv="http://xyz.mlsop.com//envelope/" FirstName>MOHANyyyy-mm-dd"

2 
gsub => ["message", "(%[a-zA-z0-9-]+)", "yyyy-mm-dd"]
  • Why do you have a % in your expression? There's no such thing in your input data.
  • [a-zA-z0-9-]+ is too broad and will match every word.
3

gsub => ["message", "^\d+-\d+-\d+$", "yyyy-mm-dd"]

changed as above, still no success.

Sample Log Message :
2018-01-10T16:58:16.925-0700 |[Container : 0]| TRACE | sent | Sent response [MOHAN1949-12-01<Last4SSN xsi:nil="true"/

4

Why do you have ^ and $ in your expression? They anchor the rest of the expression to the beginning and end of the line so you'll only replace date-like sequences if that's the only thing the input line contains.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.