Unable to remove lines from message filed

Elastic Stack Logstash
1

I am trying to remove the below lines from my jenkins build logs with mutate, gsub filter but unable to do so. Can anybody please help me in this.

Downloaded from snapshots: https://abc.com/abc.pom 16:39:44 Progress (1): 2.6 kB

2

If you want to delete events when they match a pattern then use drop.

if [message] =~ /SomePatternHere/ { drop {} }
3

I tried below filter:

if [message] = "Downloaded" { drop {} }

But didnt work for me.

4

If you want to test whether the [message] field is exactly equal to "Downloaded" then use

if [message] == "Downloaded"

If you want to test whether the [message] field contains the word "Downloaded" you can use either

if "Downloaded" in [message]

or

if [message] =~ /Downloaded/
5

Thanks Badger for your response. Below is my config file:

input {
  beats {
    port => 5044
  }
}
filter {
  prune {
    blacklist_names => [ " \e\[8mha.*\e\[0m " ]
  }
}
filter {
  mutate {
    gsub => ["message", "\e\[8mha.*\e\[0m", ""]
  }
}
filter {
  if [message] =~ /Downloaded/ {
    drop { }
  }
}

I have introduced drop filter but now i am not getting any logs on the kibana ui.

Same is with this if "Downloaded" in [message] .

6

That would suggest that all of your messages contain "Downloaded".

7

I want to remove these lines from the logs. Since they are repetitive.

Downloaded from snapshots: https://abc.com/abc.pom 16:39:44 Progress (1): 2.6 kB

I don't want the complete logs to get dropped.

Please suggest.

8

The conditional around the drop filter means only lines containing the word "Downloaded" will get dropped. Without seeing the complete data set that your beat is sending there is no way to know why that might result in no data reaching kibana.

9

It should have dropped just those lines with "Downloaded" word, but its dropping the complete logs. I tried to remove it with mutate filter. Below is the configuration:

filter {
mutate {
gsub => ["message", "Downloaded from snapshots: https://abc.com.*", ""]
}
}

This works but it is leaving blank spaces in the logs. Like there are bunch of empty lines showing between two readable lines.

Could you please suggest how can i remove empty lines from the logs because drop filter just doesn't work for me.

10

Not without seeing the actual data set you are processing. What you tried should have worked.

11

Below are the lines which are repetitve in logs:

Downloaded from snapshots: https://hc-us-east-aws-artifactory.abc.com/artifactory/maven-xyz/org/xyz/plexus/abc-components/1.1.6/abc-components-1.1.6.pom (1.9 kB at 26 kB/s)
16:39:46 Downloading from snapshots: https://hc-us-east-aws-artifactory.abc.com/artifactory/maven-xyz/org/xyz/plexus/abc/1.0.8/plexus-1.0.8.pom
16:39:46 Progress (1): 3.4/6.6 kB
Progress (1): 6.6 kB

After removing them with mutate gsub, they are leaving behind empty lines

12

I have fixed this.
filter {
mutate {
gsub => ["message", "[1]*$", ""]
}
}
One can remove empty lines with the above filter.

Thanks for the assisstance.

  1. [:space:] ↩︎

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.