# Mutate gsub regex pattern help **URL:** \ **Category:** Logstash\ **Created:** [January 4, 2021, 7:14pm UTC](https://discuss.elastic.co/t/mutate-gsub-regex-pattern-help/260122 "2021-01-04T19:14:43Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![S3l3ct3d](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s3l3ct3d/32/77965_2.png) [@S3l3ct3d](https://discuss.elastic.co/u/S3l3ct3d)\ **Post date:** [January 4, 2021, 7:14pm UTC](https://discuss.elastic.co/t/mutate-gsub-regex-pattern-help/260122/1 "2021-01-04T19:14:43Z")
I am having an issue with the regex pattern in mutate gsub function. My current filter is below ```auto filter { # Log types are "SIEM_EVENTS", "TTP_EVENTS", "AUDIT_EVENTS". mutate { gsub => ["[message]", "^<\d+>", "" ] gsub => ["[message]", "^{", "" ] gsub => ["[message]", "\\", "" ] gsub => ["[message]", "$}", "" ] } if ([message] =~ "ttpDefinition") { kv { source => "[message]" field_split => ", " value_split => ": " } mutate { add_tag => ["mimecast_ttp"] } } if ([message] =~ "auditType") { kv { source => "[message]" field_split => ", " value_split => ": " } mutate { add_tag => ["mimecast_audit"] } } else { kv { source => "[message]" field_split => "|" value_split => "=" } mutate { add_tag => ["mimecast_siem"] } } } ``` I am still currently seeing logs like this coming into Kibana ```auto { "\"messageId\"" => "\"<160978436504.10440.13591654598331832802@celery3.chartio.net>\"}\u0000", "\"userEmailAddress\"" => "someemailaddress", "\"scanResult\"" => "clean", "\"userAwarenessAction\"" => "N/A", "\"subject\"" => "[External Sender] Chartio: Red Lobster Region Level DoorDash Ops\\r\\n Report for January 04,\\r\\n 2021", "message" => "\"userEmailAddress\": \"someeamiladdress\", \"fromUserEmailAddress\": \"reports@chartio.com\", \"url\": \"https://chartio.com/doordash3/red-lobster-store-operations/report/a9ff35ed1b094b6bb8e69940088755ca.pdf\", \"ttpDefinition\": \"Default URL Protection Definition\", \"subject\": \"[External Sender] Chartio: Red Lobster Region Level DoorDash Ops\\r\\n Report for January 04,\\r\\n 2021\", \"action\": \"allow\", \"adminOverride\": \"N/A\", \"userOverride\": \"None\", \"scanResult\": \"clean\", \"category\": \"Computers & Technology\", \"sendingIp\": \"159.135.231.7\", \"userAwarenessAction\": \"N/A\", \"date\": \"2021-01-04T18:53:14+0000\", \"actions\": \"Allow\", \"route\": \"inbound\", \"creationMethod\": \"User Click\", \"emailPartsDescription\": [\"Body\"], \"messageId\": \"<160978436504.10440.13591654598331832802@celery3.chartio.net>\"}\u0000", "host" => "127.0.0.1", "\"emailPartsDescription\"" => "\"Body\"", "\"url\"" => "someurlshownhere", "\"userOverride\"" => "None", "\"sendingIp\"" => "someipaddress", "@version" => "1", "\"route\"" => "inbound", "\"category\"" => "Computers & Technology", "\"adminOverride\"" => "N/A", "\"ttpDefinition\"" => "Default URL Protection Definition", "\"action\"" => "allow", "\"actions\"" => "Allow", "\"creationMethod\"" => "User Click", "\"date\"" => "2021-01-04T18:53:14+0000", "@timestamp" => 2021-01-04T18:53:56.984Z, "tags" => [ [0] "mimecast_ttp", [1] "mimecast_siem" ], "\"fromUserEmailAddress\"" => "reports@chartio.com" } ``` I am trying remove the backslashes and double quotation marks from both sides of field:value mappings in the message but nothing I am doing is correct. Thanks for any help ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [January 4, 2021, 8:07pm UTC](https://discuss.elastic.co/t/mutate-gsub-regex-pattern-help/260122/2 "2021-01-04T20:07:52Z")
Why are you not parsing this using a json filter? > [@S3l3ct3d](#): > > ` "\"emailPartsDescription\"" => "\"Body\"",` Neither side of that contains a backslash. rubydebug is adding the backslash to escape the double quote. The actual name of the field is `"emailPartsDescription"` and its value is `"Body"`. If you want to remove double quotes use ``` gsub => ["[message]", '"', "" ] ``` When you say `gsub => ["[message]", "$}", "" ]` I think you mean `gsub => ["[message]", "}$", "" ]`, but even that will not work if you have a trailing NUL at the end of [message] ---
**Author:** ![S3l3ct3d](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s3l3ct3d/32/77965_2.png) [@S3l3ct3d](https://discuss.elastic.co/u/S3l3ct3d)\ **Post date:** [January 4, 2021, 9:31pm UTC](https://discuss.elastic.co/t/mutate-gsub-regex-pattern-help/260122/3 "2021-01-04T21:31:59Z")
> [@Badger](#): > > `["[message]", '"', "" ]` @Badger, From my knowledge I am using the json filter. So from what I am seeing in the original events that is coming from the source there are three different events that come in. SIEM events, TTP events, AUDIT events. The SIEM events come in as a json format with a " | " delimited " = " field value split style. The other two types come in as a " , " delimited " : " field value split style. Here is the "Original Events" that is coming from my source: SIEM\_EVENTS: ```auto { "message" => "<14>datetime=2020-12-28T15:19:58-0500|aCode=1zdbuOm5PCC9F9MFCTuuSg|acc=CUSA105A194|IP=104.47.70.104|Dir=Inbound|Subject=[External Sender] Order 19141384-8706668 Confirmation|MsgId=|headerFrom=orders@eat.grubhub.com|Sender=bounces+1584392-695d-kmeghan=redlobster.com@em7352.eat.grubhub.com|Rcpt=kmeghan@redlobster.com|Act=Acc|TlsVer=TLSv1.2|Cphr=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384\u0000", "@version" => "1", "@timestamp" => 2021-01-04T21:20:39.587Z, "host" => "127.0.0.1" } ``` TTP\_EVENTS ```auto { "message" => "<14>{\"userEmailAddress\": \"dhoyle@redlobster.com\", \"fromUserEmailAddress\": \"gprovins@nasrecruitment.com\", \"url\": \"https://analytics.talentegy.com/76f4f0c4-2d04-409b-9144-74babb87ceb8.js\", \"ttpDefinition\": \"Default URL Protection Definition\", \"subject\": \"[External Sender] RE: Red Lobster ACTIVATE Analytics Enhancement\", \"action\": \"allow\", \"adminOverride\": \"Allow\", \"userOverride\": \"None\", \"scanResult\": \"clean\", \"category\": \"Customer managed url allow list\", \"sendingIp\": \"40.107.70.43\", \"userAwarenessAction\": \"N/A\", \"date\": \"2021-01-04T20:58:06+0000\", \"actions\": \"Allow\", \"route\": \"inbound\", \"creationMethod\": \"User Click\", \"emailPartsDescription\": [\"Body\"], \"messageId\": \"\"}\u0000", "@timestamp" => 2021-01-04T20:58:26.698Z, "@version" => "1", "host" => "127.0.0.1" } ``` AUDIT\_EVENTS ```auto { "message" => "<14>{\"id\": \"eNoVzUsOgjAUQNG9vDEDyicFZgUdYExKNaBxRmhBCFjtR9IY9y4u4J77AS06q8TIIYPGkVQX-UrZaTI7nziW4MPU04EeV-sWfqsvrEruC-ubK4mD8iWw2puqn9LyKds5Bw9ay0czy-HvBWGEMMah70FntZGLUJ3kYhsV9ZkgPyYojbbmLZQe5QMy9P0BYhAuFw\", \"auditType\": \"Logon Authentication Failed\", \"user\": \"dkrieman@redlobster.com\", \"eventTime\": \"2021-01-04T13:57:54+0000\", \"eventInfo\": \"Failed authentication for dkrieman@redlobster.com , Date: 2021-01-04, Time: 08:57:54 GMT-05:00, IP: 194.61.53.41, Application: SMTP-MTA2, Reason: Account disabled\", \"category\": \"authentication_logs\"}\u0000", "@timestamp" => 2021-01-04T20:58:43.640Z, "@version" => "1", "host" => "127.0.0.1" } ``` I am not sure how to get this to work really. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [January 4, 2021, 10:13pm UTC](https://discuss.elastic.co/t/mutate-gsub-regex-pattern-help/260122/4 "2021-01-04T22:13:44Z")
Try ``` grok { match => { "message" => "^<%{POSINT:syslog_pri}>" } } mutate { gsub => ["message", "^<\d+>", "", "message", "\u0000", ""] } if [message] =~ /^{/ { json { source => "message" remove_field => ["message"] } } else { kv { field_split => "|" value_split => "=" remove_field => ["message"] } } ``` ---
**Author:** ![S3l3ct3d](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s3l3ct3d/32/77965_2.png) [@S3l3ct3d](https://discuss.elastic.co/u/S3l3ct3d)\ **Post date:** [January 4, 2021, 10:15pm UTC](https://discuss.elastic.co/t/mutate-gsub-regex-pattern-help/260122/5 "2021-01-04T22:15:36Z")
In place of the whole filter I already have? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [January 4, 2021, 10:26pm UTC](https://discuss.elastic.co/t/mutate-gsub-regex-pattern-help/260122/6 "2021-01-04T22:26:08Z")
If those are the only three types of messages you have then that is all you need. You can replace everything you already have. ---
**Author:** ![S3l3ct3d](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s3l3ct3d/32/77965_2.png) [@S3l3ct3d](https://discuss.elastic.co/u/S3l3ct3d)\ **Post date:** [January 4, 2021, 10:27pm UTC](https://discuss.elastic.co/t/mutate-gsub-regex-pattern-help/260122/7 "2021-01-04T22:27:12Z")
That is awesome. I am currently putting it to stdout and it looks like it is working fine. Again, thank you. I have been racking my brain over this the whole day. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 1, 2021, 10:27pm UTC](https://discuss.elastic.co/t/mutate-gsub-regex-pattern-help/260122/8 "2021-02-01T22:27:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.