Logstash parse out \r and \t entries

[motts]

Hello there,

I am having some issues with parsing out some logstash data. In my data, I have \r's and \t's that are floating around that I need to get rid of.

\tLogon ID:\t\t(0x2,0x50722C31)\r

I started trying to remove the \t's first so I made a kv

kv { 
source => [message]
trim => "\t"
}

but I found out that it's some sort of regular expression and I had to use another \ to escape the first one. I tried instead using \\t but then it removed all lower case t's and the 's instead of just the \t's :sad: I also tried \\\t but that only took out the \ and left the t's. Lastly, I tried \\\\t but it took me back to the \\t results.

If anyone could offer help, that would be appreciated.

Thanks

[Badger]

In your data do you have tab characters or literal backslash lowercase tee?

[motts]

There are literal tabs in the data. I think it's being converted into json and is shipped as pain text, which is making this complicated. I have NxLog shipping the logs to LogStash.

[Badger]

Assuming you are on UNIX, you can put a literal tab (ctrl/v ctrl/I) or carriage return (ctrl/v ctrl/m) in the logstash configuration file.

[motts]

I can try. They data doesn't show up in elasticsearch with tabs though, even if I have a blank filter just to see how the data shows up. It shows up as

\tUser Name:\t\taccount\r so there isn't literal spaces

If I click on the JSON part of the data, there is an extra \ at each one.

\\tUser Name:\\t\\taccount\\r

I'll give your suggestion a go though.

[motts]

Just tried your suggestion and it didn't seem to do any parsing. Assuming you mean pressing ctrl and v then ctrl and l it showed up as ^V^L in the config file. I am using CentOS btw

[Badger]

Tab is ctrl/eye, not ctrl/ell. I am assuming that lnext is set to ctrl/v. Does 'stty -a' show you that?

[motts]

stty -a shows lnext = ^V

I substituted the i's instead of the L's and reran the services, but the \t's and \r's are still showing

[motts]

Here is the entry, in JSON, inside elastic. Not sure if this helps or not

"User Logoff:\\r,\\r,\\tUser Name:\\tadmin\\r,\\r,\\tDomain:\\t\\tsrv-tst\\r,\\r,\\tLogon ID:\\t\\t(0x2,0x50722C31)\\r,\\r,\\tLogon Type:\\t5\\r,"

[Badger]

@motts

"User Logoff:\r,\r,\tUser Name:\tadmin\r,\r,\tDomain:\t\tsrv-tst\r,\r,\tLogon ID:\t\t(0x2,0x50722C31)\r,\r,\tLogon Type:\t5\r,"

That really suggests to me that you have backslash + r or t in your message.

mutate { gsub => [ "message", "\\[rt]", "" ] }

would cure that.

[motts]

Yeah I guess that what I was trying to say. In the original CSV, there are new lines, but NxLog seemed to replace those lines with plain-text JSON formatting for those, so the tabs were \t and the returns were \r, but as plain-text instead of code.

That worked like a charm @Badger. You are the boss!