Drop line rather than the whole message

Elastic Stack Logstash
1

hi
I am trying to drop lines which match a regex

filter {
if [message] =~ "\e\[8mha.*\e\[0m" { drop{} }
}

this above one removes the whole message , how do i drop only the line that contains the matching regex

Thanks

2

Hi,

Actually i dont think logstash is able to process or understand the concept of lines within a log. It's more likely to process a simple text string.

What do you mean by the line ? can you provide log samples or an example of what your trying to do ?

Thanks.

3

Thanks Grumo

this log is as

Started by user ^[[8mha:////4Ag0Ig80jDtqRg/RNPPpMGJ+n2VlZliPm5hsa5zI/TRRAAAAmB+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAyOEgYe/dLi1CL94pTEnMQcAONGjhrBAAAA^[[0mshantanu
Running in Durability level: MAX_SURVIVABILITY
^[[8mha:////4MU1hwwmrQi5WMhiBzYpeSIAvmxMsBgiRIwcP4r0nhFEAAAAoh+LCAAAAAAAAP9tjTEOwjAQBM8BClpKHuFItIiK1krDC0x8GCfWnbEdkooX8TX+gCESFVvtrLSa5wtWKcKBo5UdUu8otU4GP9jS5Mixv3geZcdn2TIl9igbHBs2eJyx4YwwR1SwULBGaj0nRzbDRnX6rmuvydanHMu2V1A5c4MHCFXMWcf8hSnC9jqYxPTz/BXAFEIGsfuclm8zQVqFvQAAAA==^[[0m[Pipeline] Start of Pipeline
^[[8mha:////4PvyZR0Eufn8pVijx2W+VC4vO4lzSwW+jYu+Gz5Af9ncAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycohUghExsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jduZBmjwAAAAA==^[[0m[Pipeline] node
Running on ^[[8mha:////4HFz2q8kFnRDP54rOdrme553Tm2cBbSad6RlI0bybWm1AAAAnh+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAz2EgZh/eT83ILSktQifY3cxGIgrakPAHib2iPIAAAA^[[0mJenkins in C:\Program Files (x86)\Jenkins\workspace\hello-world-pipeline
^[[8mha:////4GZ0r1Sim+oKCuXq81DZfGP5hmWUMYImNxP9daJkty7IAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0gA0xsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jfoP95RwAAAAA==^[[0m[Pipeline] {
^[[8mha:////4NJgnKgHJfLGyHX9BVJWEXk8FmH6U/cUZUfkJ68W8IFgAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0gQkxsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jc09154wAAAAA==^[[0m[Pipeline] stage
^[[8mha:////4J1SRlpdTkmcCyiVdL5f6mefhdeXHOfnyTs7jmBy+zKPAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0ggUxsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jek7ggRwAAAAA==^[[0m[Pipeline] { (Hello)
^[[8mha:////4I57OGEWTMvdmGgd0x28XIAVuBWTg5yDl6c9ouVopWC2AAAAoh+LCAAAAAAAAP9tjTEOAiEURD9rLGwtPQTbaWGsbAmNJ0AWEZb8zwLrbuWJvJp3kLiJlZNMMm+a93rDOic4UbLcG+wdZu14DKOti0+U+lugiXu6ck2YKRguzSSpM+cFJRUDS1gDKwEbgzpQdmgLbIVXD9UGhba9lFS/o4DGdQM8gYlqLiqVL8wJdvexy4Q/z18BzLEA29ce4gfg7KmOvAAAAA==^[[0m[Pipeline] echo
Hello World
^[[8mha:////4BGQRjkxyArSWBqfNvxwFoWmMQturewq5+OY+ft/+6pSAAAAoh+LCAAAAAAAAP9tjTEOAiEURD9rLGwtPQTbGRNjZUtoPAGyiLDkfxZYdytP5NW8g8RNrJxkknnTvNcb1jnBiZLl3mDvMGvHYxhtXXyi1N8CTdzTlWvCTMFwaSZJnTkvKKkYWMIaWAnYGNSBskNbYCu8eqg2KLTtpaT6HQU0rhvgCUxUc1GpfGFOsLuPXSb8ef4KYI6xADvU7j9Dg2gqvAAAAA==^[[0m[Pipeline] }
^[[8mha:////4KJLdGONlFYcajFb7HEgAxuMOBEk8OfAEAmpLY6PqYjpAAAAoh+LCAAAAAAAAP9tjTEOAiEURD9rLGwtPQRbWRhjZUtoPAGyiLDkfxZYdytP5NW8g8RNrJxkknnTvNcb1jnBiZLl3mDvMGvHYxhtXXyi1N8CTdzTlWvCTMFwaSZJnTkvKKkYWMIaWAnYGNSBskNbYCu8eqg2KLTtpaT6HQU0rhvgCUxUc1GpfGFOsLuPXSb8ef4KYI6xADvU7j9J+wGOvAAAAA==^[[0m[Pipeline] // stage
^[[8mha:////4OaUHmk6C7DGJCMQqpi+M5Ky66dEIIbY3QYtANHMdM1NAAAAox+LCAAAAAAAAP9tjTEOwjAQBDdBFLSUPMIBiQ5R0VppeIFJjHFi3QX7QlLxIr7GH4iIRMVWO9PM641lijhydKqx1HpKlVdd6N301MCxvQYeVMMXVTElDlaVdii5tqcZSxaLeVmOhcbKUhU4eXKCtW7MwxTBkCvOEid30Mh9fccTmZ7KYqJ8YYzY3Po6Mf06fwMYu06Q77aCbP8BhStF0r0AAAA=^[[0m[Pipeline] }
^[[8mha:////4NgOLXEXVVO5wLuhyp8l/YrjBFpMpCcvwaobrGA93MeeAAAAoh+LCAAAAAAAAP9tjbEOgjAURS8YB1dHP6KEOBon14bFL6hQa6F5D9uHMPlF/pr/IJHEyTvdc5bzemOdIo4cnWotdZ5S7VUfBjc/NXLsroFH1fJF1UyJg1WVHStu7GnBisViWZZjpbGxVAdOnpxgq1vzMEUw5IqzxNkdNHLf3PFEpueymChfmCJ2t6FJTL/O3wCmvhfkZSnI9h+Wl0FxvQAAAA==^[[0m[Pipeline] // node
^[[8mha:////4KY4PpR0M0VKtDOsb5ExOp9v8AJf3pk0dpEoFv+XT3uKAAAAoh+LCAAAAAAAAP9tjTESgjAQRT84FraWHiKMtI6VbYbGE0SIMZDZxWQRKk/k1byDjMxY+av/XvNeb6xTxJGjU62lzlOqverD4OanRo7dNfCoWr6omilxsKqyY8WNPS1YsVgsy3KsNDaW6sDJkxNsdWsepgiGXHGWOLuDRu6bO57I9FwWE+ULU8TuNjSJ6df5G8DU94J8Xwqy8gPQ3eZBvQAAAA==^[[0m[Pipeline] End of Pipeline
Finished: SUCCESS

and we want to remove the below

^[[8mha:////4Ag0Ig80jDtqRg/RNPPpMGJ+n2VlZliPm5hsa5zI/TRRAAAAmB+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAyOEgYe/dLi1CL94pTEnMQcAONGjhrBAAAA^[[0m

from the logs which has regex of "\e[8mha.*\e[0m"

will filebeat be able to do it if not logstash ?

4

Oh, i see there is a lot of trash data in your log.

I would try to find a way to clear the log message before it goes into logstash.

You're issue here are ANSI escape codes

Have you tried this

5

Many Many Thanks Grumo

I used this filter (had to change the pattern a bit)

filter {
  mutate {
    gsub => ["message", "\e\[8mha.*\e\[0m", ""]
  }
}

and i was able to get the right format

Started by user shantanu
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in C:\Program Files (x86)\Jenkins\workspace\hello-world-pipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Hello)
[Pipeline] echo
Hello World
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

6

I will try to work out why the original regex from the link to remove the ansi color codes didnot work and get back if i can figure that out
Thanks

1 Like
7

Perfect !

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.