Removing specific text from a log mesg

Elastic Stack Logstash
1

Is there a way to parse the log message and search for a particular string and if that is present, then to ignore/remove that string from that message?

Every log message in the log file that i am trying to parse using logstash is followed by a line (in a new line) in the below format
xxxxx (some string) : TraceLog message some_number

such messages need be ignored.
I am using the multiline codec to identify the log messages as the ones that start with |. If any message does not start with | it should get appended to previous message.
And hence the messages in the above format are getting appended to previous messages.
Is there a way that they can be removed?
. need to search the log text for the format
. if present, log text = log text minus that string.

Kindly suggest. Thank you

2

To remove parts of a string, use a mutate filter and its gsub option. To completely ignore whole events, use a drop filter that you run conditionally via an if ... { ... } block.

3

thanks Magnus,
i read about the gsub and able to achieve it partially.

The last line of my multilline log message is something like below
xxxxx (some string) : TraceLog message some_number

(some string in the beginning containing a-z A-Z 0-9 .-_: followed by a fixed string ': TraceLog message' followed by a random number)

Is there a way i can ignore this entire last line? i tried the below,

mutate {
gsub => ["Log_Text", ": TraceLog message", ""]
}
mutate {
gsub => ["Log_Text", "[0-9]*$", ""]
}

The first mutate will remove the fixed string : TraceLog message and the next one will remove the number in the end.
Even if there was no substring 'TraceLog message"in side the logtext, it is removing any string in the end!

can you plz suggest how can the reg expression be defined for this entire line, which can be used in gsub..

4

As I said, use a drop filter to completely ignore whole events.

if [message] =~ /: TraceLog message \d+/ {
  drop { }
}
5

ya, but i need to have the rest of the log_text.
im extracting the log_text using a multi-line codec and i need all the lines except the last line (in the above format)

codec => multiline {
pattern => "^|"
negate => true
what => "previous"
}

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.