Jenkins logs via logstash

Elastic Stack Logstash
1

Hi
we are trying to send Jenkins build logs logs to ES via logstash and filebeat. We have tried Jenkins logstash plugin but it doesnot work with pipelines.
Our filebeat config is like

filebeat.inputs:
- type: log
  enabled: true
  paths:
     - /jenkins-builds/jobs/*/builds/*/log

  multiline.pattern: '^[A-Z]{1}[a-z]{2} {1,2}[0-9]{1,2}, [0-9]{4} {1,2}[0-9]{1,2}:[0-9]{2}:[0-9]{2}'
  multiline.negate: true
  multiline.match: after

output.logstash:
  hosts: ["localhost:5044"]

but jenkins build logs have some metadata which we are trying to remove from the logs so we have tried adding a regex to ignore it

input {
  beats {
    port => 5044
  }
}

filter {
  prune {
    blacklist_names => [ " \e\[8mha.*\e\[0m " ]
  }
}

output {
  stdout { }
}

but the metadata is still there

`

"message": "Started by user \u001b[8mha:////4Ag0Ig80jDtqRg/RNPPpMGJ+n2VlZliPm5hsa5zI/TRRAAAAmB+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAyOEgYe/dLi1CL94pTEnMQcAONGjhrBAAAA\u001b[0mshantanu\nRunning in Durability level: MAX_SURVIVABILITY\n\u001b[8mha:////4MU1hwwmrQi5WMhiBzYpeSIAvmxMsBgiRIwcP4r0nhFEAAAAoh+LCAAAAAAAAP9tjTEOwjAQBM8BClpKHuFItIiK1krDC0x8GCfWnbEdkooX8TX+gCESFVvtrLSa5wtWKcKBo5UdUu8otU4GP9jS5Mixv3geZcdn2TIl9igbHBs2eJyx4YwwR1SwULBGaj0nRzbDRnX6rmuvydanHMu2V1A5c4MHCFXMWcf8hSnC9jqYxPTz/BXAFEIGsfuclm8zQVqFvQAAAA==\u001b[0m[Pipeline] Start of Pipeline\n\u001b[8mha:////4PvyZR0Eufn8pVijx2W+VC4vO4lzSwW+jYu+Gz5Af9ncAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycohUghExsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jduZBmjwAAAAA==\u001b[0m[Pipeline] node\nRunning on \u001b[8mha:////4HFz2q8kFnRDP54rOdrme553Tm2cBbSad6RlI0bybWm1AAAAnh+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAz2EgZh/eT83ILSktQifY3cxGIgrakPAHib2iPIAAAA\u001b[0mJenkins in C:\Program Files (x86)\Jenkins\workspace\hello-world-pipeline\n\u001b[8mha:////4GZ0r1Sim+oKCuXq81DZfGP5hmWUMYImNxP9daJkty7IAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0gA0xsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jfoP95RwAAAAA==\u001b[0m[Pipeline] {\n\u001b[8mha:////4NJgnKgHJfLGyHX9BVJWEXk8FmH6U/cUZUfkJ68W8IFgAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0gQkxsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jc09154wAAAAA==\u001b[0m[Pipeline] stage\n\u001b[8mha:////4J1SRlpdTkmcCyiVdL5f6mefhdeXHOfnyTs7jmBy+zKPAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0ggUxsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jek7ggRwAAAAA==\u001b[0m[Pipeline] { (Hello)\n\u001b[8mha:////4I57OGEWTMvdmGgd0x28XIAVuBWTg5yDl6c9ouVopWC2AAAAoh+LCAAAAAAAAP9tjTEOAiEURD9rLGwtPQTbaWGsbAmNJ0AWEZb8zwLrbuWJvJp3kLiJlZNMMm+a93rDOic4UbLcG+wdZu14DKOti0+U+lugiXu6ck2YKRguzSSpM+cFJRUDS1gDKwEbgzpQdmgLbIVXD9UGhba9lFS/o4DGdQM8gYlqLiqVL8wJdvexy4Q/z18BzLEA29ce4gfg7KmOvAAAAA==\u001b[0m[Pipeline] echo\nHello World\n\u001b[8mha:////4BGQRjkxyArSWBqfNvxwFoWmMQturewq5+OY+ft/+6pSAAAAoh+LCAAAAAAAAP9tjTEOAiEURD9rLGwtPQTbGRNjZUtoPAGyiLDkfxZYdytP5NW8g8RNrJxkknnTvNcb1jnBiZLl3mDvMGvHYxhtXXyi1N8CTdzTlWvCTMFwaSZJnTkvKKkYWMIaWAnYGNSBskNbYCu8eqg2KLTtpaT6HQU0rhvgCUxUc1GpfGFOsLuPXSb8ef4KYI6xADvU7j9Dg2gqvAAAAA==\u001b[0m[Pipeline] }\n\u001b[8mha:////4KJLdGONlFYcajFb7HEgAxuMOBEk8OfAEAmpLY6PqYjpAAAAoh+LCAAAAAAAAP9tjTEOAiEURD9rLGwtPQRbWRhjZUtoPAGyiLDkfxZYdytP5NW8g8RNrJxkknnTvNcb1jnBiZLl3mDvMGvHYxhtXXyi1N8CTdzTlWvCTMFwaSZJnTkvKKkYWMIaWAnYGNSBskNbYCu8eqg2KLTtpaT6HQU0rhvgCUxUc1GpfGFOsLuPXSb8ef4KYI6xADvU7j9J+wGOvAAAAA==\u001b[0m[Pipeline] // stage\n\u001b[8mha:////4OaUHmk6C7DGJCMQqpi+M5Ky66dEIIbY3QYtANHMdM1NAAAAox+LCAAAAAAAAP9tjTEOwjAQBDdBFLSUPMIBiQ5R0VppeIFJjHFi3QX7QlLxIr7GH4iIRMVWO9PM641lijhydKqx1HpKlVdd6N301MCxvQYeVMMXVTElDlaVdii5tqcZSxaLeVmOhcbKUhU4eXKCtW7MwxTBkCvOEid30Mh9fccTmZ7KYqJ8YYzY3Po6Mf06fwMYu06Q77aCbP8BhStF0r0AAAA=\u001b[0m[Pipeline] }\n\u001b[8mha:////4NgOLXEXVVO5wLuhyp8l/YrjBFpMpCcvwaobrGA93MeeAAAAoh+LCAAAAAAAAP9tjbEOgjAURS8YB1dHP6KEOBon14bFL6hQa6F5D9uHMPlF/pr/IJHEyTvdc5bzemOdIo4cnWotdZ5S7VUfBjc/NXLsroFH1fJF1UyJg1WVHStu7GnBisViWZZjpbGxVAdOnpxgq1vzMEUw5IqzxNkdNHLf3PFEpueymChfmCJ2t6FJTL/O3wCmvhfkZSnI9h+Wl0FxvQAAAA==\u001b[0m[Pipeline] // node\n\u001b[8mha:////4KY4PpR0M0VKtDOsb5ExOp9v8AJf3pk0dpEoFv+XT3uKAAAAoh+LCAAAAAAAAP9tjTESgjAQRT84FraWHiKMtI6VbYbGE0SIMZDZxWQRKk/k1byDjMxY+av/XvNeb6xTxJGjU62lzlOqverD4OanRo7dNfCoWr6omilxsKqyY8WNPS1YsVgsy3KsNDaW6sDJkxNsdWsepgiGXHGWOLuDRu6bO57I9FwWE+ULU8TuNjSJ6df5G8DU94J8Xwqy8gPQ3eZBvQAAAA==\u001b[0m[Pipeline] End of Pipeline\nFinished: SUCCESS",

is there a way to remove the metadata from jenkins logs

`

2

the build logs are like this

Started by user ^[[8mha:////4Ag0Ig80jDtqRg/RNPPpMGJ+n2VlZliPm5hsa5zI/TRRAAAAmB+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAyOEgYe/dLi1CL94pTEnMQcAONGjhrBAAAA^[[0mshantanu
Running in Durability level: MAX_SURVIVABILITY
^[[8mha:////4MU1hwwmrQi5WMhiBzYpeSIAvmxMsBgiRIwcP4r0nhFEAAAAoh+LCAAAAAAAAP9tjTEOwjAQBM8BClpKHuFItIiK1krDC0x8GCfWnbEdkooX8TX+gCESFVvtrLSa5wtWKcKBo5UdUu8otU4GP9jS5Mixv3geZcdn2TIl9igbHBs2eJyx4YwwR1SwULBGaj0nRzbDRnX6rmuvydanHMu2V1A5c4MHCFXMWcf8hSnC9jqYxPTz/BXAFEIGsfuclm8zQVqFvQAAAA==^[[0m[Pipeline] Start of Pipeline
^[[8mha:////4PvyZR0Eufn8pVijx2W+VC4vO4lzSwW+jYu+Gz5Af9ncAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycohUghExsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jduZBmjwAAAAA==^[[0m[Pipeline] node
Running on ^[[8mha:////4HFz2q8kFnRDP54rOdrme553Tm2cBbSad6RlI0bybWm1AAAAnh+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAz2EgZh/eT83ILSktQifY3cxGIgrakPAHib2iPIAAAA^[[0mJenkins in C:\Program Files (x86)\Jenkins\workspace\hello-world-pipeline
^[[8mha:////4GZ0r1Sim+oKCuXq81DZfGP5hmWUMYImNxP9daJkty7IAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0gA0xsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jfoP95RwAAAAA==^[[0m[Pipeline] {
^[[8mha:////4NJgnKgHJfLGyHX9BVJWEXk8FmH6U/cUZUfkJ68W8IFgAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0gQkxsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jc09154wAAAAA==^[[0m[Pipeline] stage
^[[8mha:////4J1SRlpdTkmcCyiVdL5f6mefhdeXHOfnyTs7jmBy+zKPAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0ggUxsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jek7ggRwAAAAA==^[[0m[Pipeline] { (Hello)
^[[8mha:////4I57OGEWTMvdmGgd0x28XIAVuBWTg5yDl6c9ouVopWC2AAAAoh+LCAAAAAAAAP9tjTEOAiEURD9rLGwtPQTbaWGsbAmNJ0AWEZb8zwLrbuWJvJp3kLiJlZNMMm+a93rDOic4UbLcG+wdZu14DKOti0+U+lugiXu6ck2YKRguzSSpM+cFJRUDS1gDKwEbgzpQdmgLbIVXD9UGhba9lFS/o4DGdQM8gYlqLiqVL8wJdvexy4Q/z18BzLEA29ce4gfg7KmOvAAAAA==^[[0m[Pipeline] echo
Hello World
^[[8mha:////4BGQRjkxyArSWBqfNvxwFoWmMQturewq5+OY+ft/+6pSAAAAoh+LCAAAAAAAAP9tjTEOAiEURD9rLGwtPQTbGRNjZUtoPAGyiLDkfxZYdytP5NW8g8RNrJxkknnTvNcb1jnBiZLl3mDvMGvHYxhtXXyi1N8CTdzTlWvCTMFwaSZJnTkvKKkYWMIaWAnYGNSBskNbYCu8eqg2KLTtpaT6HQU0rhvgCUxUc1GpfGFOsLuPXSb8ef4KYI6xADvU7j9Dg2gqvAAAAA==^[[0m[Pipeline] }
^[[8mha:////4KJLdGONlFYcajFb7HEgAxuMOBEk8OfAEAmpLY6PqYjpAAAAoh+LCAAAAAAAAP9tjTEOAiEURD9rLGwtPQRbWRhjZUtoPAGyiLDkfxZYdytP5NW8g8RNrJxkknnTvNcb1jnBiZLl3mDvMGvHYxhtXXyi1N8CTdzTlWvCTMFwaSZJnTkvKKkYWMIaWAnYGNSBskNbYCu8eqg2KLTtpaT6HQU0rhvgCUxUc1GpfGFOsLuPXSb8ef4KYI6xADvU7j9J+wGOvAAAAA==^[[0m[Pipeline] // stage
^[[8mha:////4OaUHmk6C7DGJCMQqpi+M5Ky66dEIIbY3QYtANHMdM1NAAAAox+LCAAAAAAAAP9tjTEOwjAQBDdBFLSUPMIBiQ5R0VppeIFJjHFi3QX7QlLxIr7GH4iIRMVWO9PM641lijhydKqx1HpKlVdd6N301MCxvQYeVMMXVTElDlaVdii5tqcZSxaLeVmOhcbKUhU4eXKCtW7MwxTBkCvOEid30Mh9fccTmZ7KYqJ8YYzY3Po6Mf06fwMYu06Q77aCbP8BhStF0r0AAAA=^[[0m[Pipeline] }
^[[8mha:////4NgOLXEXVVO5wLuhyp8l/YrjBFpMpCcvwaobrGA93MeeAAAAoh+LCAAAAAAAAP9tjbEOgjAURS8YB1dHP6KEOBon14bFL6hQa6F5D9uHMPlF/pr/IJHEyTvdc5bzemOdIo4cnWotdZ5S7VUfBjc/NXLsroFH1fJF1UyJg1WVHStu7GnBisViWZZjpbGxVAdOnpxgq1vzMEUw5IqzxNkdNHLf3PFEpueymChfmCJ2t6FJTL/O3wCmvhfkZSnI9h+Wl0FxvQAAAA==^[[0m[Pipeline] // node
^[[8mha:////4KY4PpR0M0VKtDOsb5ExOp9v8AJf3pk0dpEoFv+XT3uKAAAAoh+LCAAAAAAAAP9tjTESgjAQRT84FraWHiKMtI6VbYbGE0SIMZDZxWQRKk/k1byDjMxY+av/XvNeb6xTxJGjU62lzlOqverD4OanRo7dNfCoWr6omilxsKqyY8WNPS1YsVgsy3KsNDaW6sDJkxNsdWsepgiGXHGWOLuDRu6bO57I9FwWE+ULU8TuNjSJ6df5G8DU94J8Xwqy8gPQ3eZBvQAAAA==^[[0m[Pipeline] End of Pipeline
Finished: SUCCESS

3

or simply has anyone got the jenkins logs in logstash working
Thanks

4

this one fixed it

filter {
  mutate {
    gsub => ["message", "\e\[8mha.*\e\[0m", ""]
  }
}
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.