Remove \n and \t from message

[ArunANayagam]

Hi,

I am trying to ingest Java exceptions into Elasticsearch via logstash.

In Kibana, they show up like this,

\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)\n\tat org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799)\n\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)\n\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861)\n\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455)\n\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tat java.lang.Thread.run(Thread.java:748)\n

Is there a way to remove the \n and \t and make it more readable?

I did try,

mutate { gsub => [ "message", "[\n]", "" ] }

That didn't help. The message remained the same.

Appreciate your help on this.

Thanks,
Arun

[Badger]

    mutate { gsub => [ "message", "[
    ]+", " " ] }

That's literal newline and tab in the config.

[ArunANayagam]

Hi Badger,

Thanks for the response, unfortunately that doesn't help.

Here's my filter

filter {
    mutate { gsub => [ "message", "[
    ]+", "" ] }
    json {
        source => "message"
    }
}

Thanks,
Arun

[Badger]

What does the message look like if you add

output { stdout { codec => rubydebug } }

and what do you want it to look like.

[ArunANayagam]

Hi Badger,

I just realised that the actual spaces/tabs are getting replaced but not the literal \n\t.

Like here,

Error:FPP-2Unabletoreaduserfromdatabase:8098791524808184-tstff:StackTrace:org.springframework.web.client.HttpClientErrorException:403\n\tatorg.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:63)\n\tatorg.springframework

So the first line should be,
"Unable to read user from database" -- but the spaces are eaten up.
But those \n\t are still present as is.

I use java's org.apache.commons.lang3.exception.ExceptionUtils.getStackTrace(e) to convert multi-line stack trace to string and that's introducing those \n and \t 's

Within elasticsearch, this is exactly how the document looks like,

{
    "_index": "dev-fps-logs-2018.08.01",
    "_type": "doc",
    "_id": "Ss2r9mQBXavQlNSUIlgG",
    "_version": 1,
    "found": true,
    "_source": {
        "msg": "\" :Error: FPS-2 Unable to read user from database: 8098791524808184-tst22 :StackTrace: org.springframework.web.client.HttpClientErrorException: 403 \\n\\tat org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:63)\\n\\tat org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:700)\\n\\tat org.springframework

Thanks,
Arun

[ArunANayagam]

Hi Badger,

I managed to get it working, I had to do this. Really appreciate your help on this.

filter {
    json {
        source => "message"
    }
    mutate {
        remove_field => ["message"]
    }
    mutate { gsub => [ "msg", "\\n\\t", " " ] }
}