Hi All,
i have a scenirio where my Jenkins is pushing builld logs to logstash line by line. the issue here is the data reccived by the ELK has the meta data which is common to all the logs reccived by logstash for a single build in Jenkins. is there a way where we can remove all the duplicate meta data for a single build of jenkins in logstash.
below is example. my Jenkins console output has 50 lines. so it sends 50 entries to the logstash. below is the output from logstash
entry 1
{"data":{"id":"33","result":"SUCCESS","projectName":"ELK progress","fullProjectName":"ELK progress","displayName":"#33","fullDisplayName":"ELK progress #33","url":"job/ELK%20progress/33/","buildHost":"Jenkins","buildLabel":"master","buildNum":33,"buildDuration":0,"rootProjectName":"ELK progress","rootFullProjectName":"ELK progress","rootProjectDisplayName":"#33","rootBuildNum":33,"buildVariables":{"PROCESSOR_LEVEL":"6","RUN_DISPLAY_URL":"http://10.128.200.4:8080/job/ELK%20progress/33/display/redirect","FP_NO_HOST_CHECK":"NO","HUDSON_HOME":"C:\\Program Files (x86)\Jenkins","RUN_CHANGES_DISPLAY_URL":"http://10.128.200.4:8080/job/ELK%20progress/33/display/redirect?GenuineIntel","JOB_DISPLAY_URL":"http://10.128.200.4:8080/job/ELK%20progress/display/redirect","WINSW_EXECUTABLE":"C:\\Program Files (x86)\Jenkins\jenkins.exe","ProgramFiles":"C:\Program Files (x86)","PUBLIC":"C:\Users\Public","windir":"C:\WINDOWS","LOCALAPPDATA":"C:\WINDOWS\system32\config\systemprofile\AppData\Local","USERDOMAIN":"WORKGROUP","BUILD_ID":"33","BASE":"C:\Program Files (x86)\Jenkins","JOB_NAME":"ELK progress","APPDATA":"C:\WINDOWS\system32\config\systemprofile\AppData\Roaming","NODE_LABELS":"master","HUDSON_URL":"http://10.128.200.4:8080/","windows_tracing_logfile":"C:\\BVTBin\\Tests\\installpackage\\csilogfile.log","CommonProgramFiles":"C:\\Program Files (x86)\Common Files","Path":"C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files\Git\cmd;C:\WINDOWS\System32\OpenSSH\;C:\WINDOWS\system32\config\systemprofile\AppData\Local\Microsoft\WindowsApps","OS":"Windows_NT","windows_tracing_flags":"3","COMPUTERNAME":"WIN10X64BASE","EXECUTOR_NUMBER":"0","HUDSON_SERVER_COOKIE":"7a414e699751cd17","JOB_BASE_NAME":"ELK progress","PROCESSOR_REVISION":"4f00","CLASSPATH":"","CommonProgramW6432":"C:\Program Files\Common Files","ComSpec":"C:\WINDOWS\system32\cmd.exe","BUILD_NUMBER":"33","SystemRoot":"C:\WINDOWS","TEMP":"C:\WINDOWS\TEMP","BUILD_URL":"http://10.128.200.4:8080/job/ELK%20progress/33/","USERPROFILE":"C:\\WINDOWS\\system32\\config\\systemprofile","TMP":"C:\\WINDOWS\\TEMP","CommonProgramFiles(x86)":"C:\\Program Files (x86)\Common Files","NUMBER_OF_PROCESSORS":"4"},"sensitiveBuildVariables":}
**message":["There is no public Hudson instance configured for this project"]
**,"source":"jenkins","source_host":"http://10.128.200.4:8080/","@buildTimestamp":"2019-01-04T10:21:40-0600","@timestamp":"2019-01-04T10:21:40-0600","@version":1}
entry 2
"data":{"id":"33","result":"SUCCESS","projectName":"ELK progress","fullProjectName":"ELK progress","displayName":"#33","fullDisplayName":"ELK progress #33","url":"job/ELK%20progress/33/","buildHost":"Jenkins","buildLabel":"master","buildNum":33,"buildDuration":0,"rootProjectName":"ELK progress","rootFullProjectName":"ELK progress","rootProjectDisplayName":"#33","rootBuildNum":33,"buildVariables":{"PROCESSOR_LEVEL":"6","RUN_DISPLAY_URL":"http://10.128.200.4:8080/job/ELK%20progress/33/display/redirect","FP_NO_HOST_CHECK":"NO","HUDSON_HOME":"C:\\Program Files (x86)\Jenkins","RUN_CHANGES_DISPLAY_URL":"http://10.128.200.4:8080/job/ELK%20progress/33/display/redirect?GenuineIntel","JOB_DISPLAY_URL":"http://10.128.200.4:8080/job/ELK%20progress/display/redirect","WINSW_EXECUTABLE":"C:\\Program Files (x86)\Jenkins\jenkins.exe","ProgramFiles":"C:\Program Files (x86)","PUBLIC":"C:\Users\Public","windir":"C:\WINDOWS","LOCALAPPDATA":"C:\WINDOWS\system32\config\systemprofile\AppData\Local","USERDOMAIN":"WORKGROUP","BUILD_ID":"33","BASE":"C:\Program Files (x86)\Jenkins","JOB_NAME":"ELK progress","APPDATA":"C:\WINDOWS\system32\config\systemprofile\AppData\Roaming","NODE_LABELS":"master","HUDSON_URL":"http://10.128.200.4:8080/","windows_tracing_logfile":"C:\\BVTBin\\Tests\\installpackage\\csilogfile.log","CommonProgramFiles":"C:\\Program Files (x86)\Common Files","Path":"C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files\Git\cmd;C:\WINDOWS\System32\OpenSSH\;C:\WINDOWS\system32\config\systemprofile\AppData\Local\Microsoft\WindowsApps","OS":"Windows_NT","windows_tracing_flags":"3","COMPUTERNAME":"WIN10X64BASE","EXECUTOR_NUMBER":"0","HUDSON_SERVER_COOKIE":"7a414e699751cd17","JOB_BASE_NAME":"ELK progress","PROCESSOR_REVISION":"4f00","CLASSPATH":"","CommonProgramW6432":"C:\Program Files\Common Files","ComSpec":"C:\WINDOWS\system32\cmd.exe","BUILD_NUMBER":"33","SystemRoot":"C:\WINDOWS","TEMP":"C:\WINDOWS\TEMP","BUILD_URL":"http://10.128.200.4:8080/job/ELK%20progress/33/","USERPROFILE":"C:\\WINDOWS\\system32\\config\\systemprofile","TMP":"C:\\WINDOWS\\TEMP","CommonProgramFiles(x86)":"C:\\Program Files (x86)\Common Files","NUMBER_OF_PROCESSORS":"4"},"sensitiveBuildVariables":},"*
message":["Finished: SUCCESS"],*
"source":"jenkins","source_host":"http://10.128.200.4:8080/","@buildTimestamp":"2019-01-04T10:21:40-0600","@timestamp":"2019-01-04T10:21:40-0600","@version":1}
you will notice that only messge is differnent rest all are metadata which is same. i would like to keep only one meta-data for all the build which has same build number