# Sensitive information parsing

**URL:** <https://discuss.elastic.co/t/sensitive-information-parsing/120685>\
**Category:** Logstash\
**Created:** [February 20, 2018, 4:40pm UTC](https://discuss.elastic.co/t/sensitive-information-parsing/120685 "2018-02-20T16:40:29Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![mohank44](https://avatars.discourse-cdn.com/v4/letter/m/8e7dd6/32.png) [@mohank44](https://discuss.elastic.co/u/mohank44)\
**Post date:** [February 20, 2018, 4:40pm UTC](https://discuss.elastic.co/t/sensitive-information-parsing/120685/1 "2018-02-20T16:40:29Z")

</div>

Hello All,

I am trying to mask a sensitive information from a log message.

I have given the sample log file and the config file as below. i was trying to do this using gsub with no success. Please review and advise at the earliest.

**Sample Log Message :**

2018-01-10T16:58:16.925-0700 |[WebContainer : 0]| TRACE | sent | Sent response [\<soapenv:Envelope xmlns:soapenv="[http://xyz.mlsop.com//envelope/](http://xyz.mlsop.com//envelope/)" FirstName\>MOHAN1949-12-01

**Output from Logstash :**

```
      "Type" => "TRACE",
      "Task" => "sent",
"@timestamp" => 2018-02-20T16:21:35.948Z,
  "Messsage" => " CO Region Expiry days not added to the expiration Date",
  "@version" => "1",
      "host" => "oc1008401175.ibm.com",
   "message" => "2018-01-10T16:58:16.925-0700 |[WebContainer : 0]| TRACE | sent | Sent response [<soapenv:Envelope xmlns:soapenv="http://xyz.mlsop.com//envelope/" FirstName><MiddleName></MiddleName><LastName>MOHAN</LastName><DateOfBirth>1949-12-01</DateOfBirth>",
      "Date" => "2018-01-10T16:58:16.925-0700",
 "MsgSource" => "[WebContainer : 6]",

```

**Config file:**

input {  
file {  
path =\> "/home/mohank44/Data/6LogAnalysis/PSI/PSItest5.log"  
#type =\> "LOG"  
start\_position =\> "beginning"  
codec =\> multiline  
{  
pattern =\> "^\A%{TIMESTAMP\_ISO8601}%{SPACE}|%{SYSLOG5424SD}|%{SPACE}%{WORD}"  
negate =\> true  
what =\> previous  
} }  
}  
filter{  
mutate  
{  
gsub =\> ["message", "\n", ""]  
gsub =\> ["message", "\r", ""]  
gsub =\> ["message", "\t", ""]  
gsub =\> ["message", "(%[a-zA-z0-9-]+)", "yyyy-mm-dd"]  
}  
grok {  
match =\> ["message", "\A%{TIMESTAMP\_ISO8601:Date}%{SPACE}|%{SYSLOG5424SD:MsgSource}|%{SPACE}%{WORD:Type}%{SPACE}|%{SPACE}%{WORD:Task}%{SPACE}|%{GREEDYDATA:Messsage}"]  
overwrite =\> ["message"]  
}  
date {  
match =\> ["Date","yyyy-MM-dd'T'HH:mm:ss.SSS-ZZZZ"]  
target =\> "Date"  
}

}  
output  
{  
stdout {codec =\> rubydebug}  
#stdout{}  
elasticsearch  
{  
hosts =\> "localhost"  
index =\> "log-psi-index3"  
}  
}  
**Expected Output :**

"Message" =\> "2018-01-10T16:58:16.925-0700 |[WebContainer : 0]| TRACE | sent | Sent response [\<soapenv:Envelope xmlns:soapenv="[http://xyz.mlsop.com//envelope/](http://xyz.mlsop.com//envelope/)" FirstName\>MOHANyyyy-mm-dd"

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 20, 2018, 8:13pm UTC](https://discuss.elastic.co/t/sensitive-information-parsing/120685/2 "2018-02-20T20:13:21Z")

</div>

> ```
> gsub => ["message", "(%[a-zA-z0-9-]+)", "yyyy-mm-dd"]
> 
> ```

- Why do you have a % in your expression? There's no such thing in your input data.
- `[a-zA-z0-9-]+` is too broad and will match every word.

---

<div class="post-metadata">

**Author:** ![mohank44](https://avatars.discourse-cdn.com/v4/letter/m/8e7dd6/32.png) [@mohank44](https://discuss.elastic.co/u/mohank44)\
**Post date:** [February 21, 2018, 4:42am UTC](https://discuss.elastic.co/t/sensitive-information-parsing/120685/3 "2018-02-21T04:42:30Z")

</div>

gsub =\> ["message", "^\d+-\d+-\d+$", "yyyy-mm-dd"]

changed as above, still no success.

**Sample Log Message :**  
2018-01-10T16:58:16.925-0700 |[Container : 0]| TRACE | sent | Sent response [MOHAN1949-12-01\<Last4SSN xsi:nil="true"/

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 21, 2018, 5:33am UTC](https://discuss.elastic.co/t/sensitive-information-parsing/120685/4 "2018-02-21T05:33:32Z")

</div>

Why do you have `^` and `$` in your expression? They anchor the rest of the expression to the beginning and end of the line so you'll only replace date-like sequences if that's the only thing the input line contains.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 21, 2018, 5:33am UTC](https://discuss.elastic.co/t/sensitive-information-parsing/120685/5 "2018-03-21T05:33:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.