Removing Metadata from logs in logstash

Hi All,

i have a scenirio where my Jenkins is pushing builld logs to logstash line by line. the issue here is the data reccived by the ELK has the meta data which is common to all the logs reccived by logstash for a single build in Jenkins. is there a way where we can remove all the duplicate meta data for a single build of jenkins in logstash.
below is example. my Jenkins console output has 50 lines. so it sends 50 entries to the logstash. below is the output from logstash

entry 1

{"data":{"id":"33","result":"SUCCESS","projectName":"ELK progress","fullProjectName":"ELK progress","displayName":"#33","fullDisplayName":"ELK progress #33","url":"job/ELK%20progress/33/","buildHost":"Jenkins","buildLabel":"master","buildNum":33,"buildDuration":0,"rootProjectName":"ELK progress","rootFullProjectName":"ELK progress","rootProjectDisplayName":"#33","rootBuildNum":33,"buildVariables":{"PROCESSOR_LEVEL":"6","RUN_DISPLAY_URL":"http://10.128.200.4:8080/job/ELK%20progress/33/display/redirect","FP_NO_HOST_CHECK":"NO","HUDSON_HOME":"C:\Program Files (x86)\Jenkins","RUN_CHANGES_DISPLAY_URL":"http://10.128.200.4:8080/job/ELK%20progress/33/display/redirect?GenuineIntel","JOB_DISPLAY_URL":"http://10.128.200.4:8080/job/ELK%20progress/display/redirect","WINSW_EXECUTABLE":"C:\Program Files (x86)\Jenkins\jenkins.exe","ProgramFiles":"C:\Program Files (x86)","PUBLIC":"C:\Users\Public","windir":"C:\WINDOWS","LOCALAPPDATA":"C:\WINDOWS\system32\config\systemprofile\AppData\Local","USERDOMAIN":"WORKGROUP","BUILD_ID":"33","BASE":"C:\Program Files (x86)\Jenkins","JOB_NAME":"ELK progress","APPDATA":"C:\WINDOWS\system32\config\systemprofile\AppData\Roaming","NODE_LABELS":"master","HUDSON_URL":"http://10.128.200.4:8080/","windows_tracing_logfile":"C:\BVTBin\Tests\installpackage\csilogfile.log","CommonProgramFiles":"C:\Program Files (x86)\Common Files","Path":"C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files\Git\cmd;C:\WINDOWS\System32\OpenSSH\;C:\WINDOWS\system32\config\systemprofile\AppData\Local\Microsoft\WindowsApps","OS":"Windows_NT","windows_tracing_flags":"3","COMPUTERNAME":"WIN10X64BASE","EXECUTOR_NUMBER":"0","HUDSON_SERVER_COOKIE":"7a414e699751cd17","JOB_BASE_NAME":"ELK progress","PROCESSOR_REVISION":"4f00","CLASSPATH":"","CommonProgramW6432":"C:\Program Files\Common Files","ComSpec":"C:\WINDOWS\system32\cmd.exe","BUILD_NUMBER":"33","SystemRoot":"C:\WINDOWS","TEMP":"C:\WINDOWS\TEMP","BUILD_URL":"http://10.128.200.4:8080/job/ELK%20progress/33/","USERPROFILE":"C:\WINDOWS\system32\config\systemprofile","TMP":"C:\WINDOWS\TEMP","CommonProgramFiles(x86)":"C:\Program Files (x86)\Common Files","NUMBER_OF_PROCESSORS":"4"},"sensitiveBuildVariables":}

**message":["There is no public Hudson instance configured for this project"]

**,"source":"jenkins","source_host":"http://10.128.200.4:8080/","@buildTimestamp":"2019-01-04T10:21:40-0600","@timestamp":"2019-01-04T10:21:40-0600","@version":1}

entry 2


"data":{"id":"33","result":"SUCCESS","projectName":"ELK progress","fullProjectName":"ELK progress","displayName":"#33","fullDisplayName":"ELK progress #33","url":"job/ELK%20progress/33/","buildHost":"Jenkins","buildLabel":"master","buildNum":33,"buildDuration":0,"rootProjectName":"ELK progress","rootFullProjectName":"ELK progress","rootProjectDisplayName":"#33","rootBuildNum":33,"buildVariables":{"PROCESSOR_LEVEL":"6","RUN_DISPLAY_URL":"http://10.128.200.4:8080/job/ELK%20progress/33/display/redirect","FP_NO_HOST_CHECK":"NO","HUDSON_HOME":"C:\Program Files (x86)\Jenkins","RUN_CHANGES_DISPLAY_URL":"http://10.128.200.4:8080/job/ELK%20progress/33/display/redirect?GenuineIntel","JOB_DISPLAY_URL":"http://10.128.200.4:8080/job/ELK%20progress/display/redirect","WINSW_EXECUTABLE":"C:\Program Files (x86)\Jenkins\jenkins.exe","ProgramFiles":"C:\Program Files (x86)","PUBLIC":"C:\Users\Public","windir":"C:\WINDOWS","LOCALAPPDATA":"C:\WINDOWS\system32\config\systemprofile\AppData\Local","USERDOMAIN":"WORKGROUP","BUILD_ID":"33","BASE":"C:\Program Files (x86)\Jenkins","JOB_NAME":"ELK progress","APPDATA":"C:\WINDOWS\system32\config\systemprofile\AppData\Roaming","NODE_LABELS":"master","HUDSON_URL":"http://10.128.200.4:8080/","windows_tracing_logfile":"C:\BVTBin\Tests\installpackage\csilogfile.log","CommonProgramFiles":"C:\Program Files (x86)\Common Files","Path":"C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files\Git\cmd;C:\WINDOWS\System32\OpenSSH\;C:\WINDOWS\system32\config\systemprofile\AppData\Local\Microsoft\WindowsApps","OS":"Windows_NT","windows_tracing_flags":"3","COMPUTERNAME":"WIN10X64BASE","EXECUTOR_NUMBER":"0","HUDSON_SERVER_COOKIE":"7a414e699751cd17","JOB_BASE_NAME":"ELK progress","PROCESSOR_REVISION":"4f00","CLASSPATH":"","CommonProgramW6432":"C:\Program Files\Common Files","ComSpec":"C:\WINDOWS\system32\cmd.exe","BUILD_NUMBER":"33","SystemRoot":"C:\WINDOWS","TEMP":"C:\WINDOWS\TEMP","BUILD_URL":"http://10.128.200.4:8080/job/ELK%20progress/33/","USERPROFILE":"C:\WINDOWS\system32\config\systemprofile","TMP":"C:\WINDOWS\TEMP","CommonProgramFiles(x86)":"C:\Program Files (x86)\Common Files","NUMBER_OF_PROCESSORS":"4"},"sensitiveBuildVariables":},"*

message":["Finished: SUCCESS"],*

"source":"jenkins","source_host":"http://10.128.200.4:8080/","@buildTimestamp":"2019-01-04T10:21:40-0600","@timestamp":"2019-01-04T10:21:40-0600","@version":1}

you will notice that only messge is differnent rest all are metadata which is same. i would like to keep only one meta-data for all the build which has same build number

If you want to aggregate all the messages for a build into a single event then an aggregate filter might be able to do it. It is impossible to read you example data. You need to quote it (select the text and click on the </> in the bar above the edit window).

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.