# Removing Metadata from logs in logstash
**URL:** https://discuss.elastic.co/t/removing-metadata-from-logs-in-logstash/162994
**Category:** Logstash
**Created:** [January 4, 2019, 5:50pm UTC](https://discuss.elastic.co/t/removing-metadata-from-logs-in-logstash/162994 "2019-01-04T17:50:40Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@mayanks8](https://discuss.elastic.co/u/mayanks8)
#### Post date: [January 4, 2019, 5:50pm UTC](https://discuss.elastic.co/t/removing-metadata-from-logs-in-logstash/162994/1 "2019-01-04T17:50:40Z")
Hi All,
i have a scenirio where my Jenkins is pushing builld logs to logstash line by line. the issue here is the data reccived by the ELK has the meta data which is common to all the logs reccived by logstash for a single build in Jenkins. is there a way where we can remove all the duplicate meta data for a single build of jenkins in logstash.
below is example. my Jenkins console output has 50 lines. so it sends 50 entries to the logstash. below is the output from logstash
## entry 1
{"data":{"id":"33","result":"SUCCESS","projectName":"ELK progress","fullProjectName":"ELK progress","displayName":"#33","fullDisplayName":"ELK progress #33","url":"job/ELK%20progress/33/","buildHost":"Jenkins","buildLabel":"master","buildNum":33,"buildDuration":0,"rootProjectName":"ELK progress","rootFullProjectName":"ELK progress","rootProjectDisplayName":"#33","rootBuildNum":33,"buildVariables":{"PROCESSOR\_LEVEL":"6","RUN\_DISPLAY\_URL":"[http://10.128.200.4:8080/job/ELK%20progress/33/display/redirect","FP\_NO\_HOST\_CHECK":"NO","HUDSON\_HOME":"C:\\Program](http://10.128.200.4:8080/job/ELK%20progress/33/display/redirect%22,%22FP_NO_HOST_CHECK%22:%22NO%22,%22HUDSON_HOME%22:%22C:%5C%5CProgram) Files (x86)\Jenkins","RUN\_CHANGES\_DISPLAY\_URL":"[http://10.128.200.4:8080/job/ELK%20progress/33/display/redirect?GenuineIntel","JOB\_DISPLAY\_URL":"http://10.128.200.4:8080/job/ELK%20progress/display/redirect","WINSW\_EXECUTABLE":"C:\\Program](http://10.128.200.4:8080/job/ELK%20progress/33/display/redirect?GenuineIntel%22,%22JOB_DISPLAY_URL%22:%22http://10.128.200.4:8080/job/ELK%20progress/display/redirect%22,%22WINSW_EXECUTABLE%22:%22C:%5C%5CProgram) Files (x86)\Jenkins\jenkins.exe","ProgramFiles":"C:\Program Files (x86)","PUBLIC":"C:\Users\Public","windir":"C:\WINDOWS","LOCALAPPDATA":"C:\WINDOWS\system32\config\systemprofile\AppData\Local","USERDOMAIN":"WORKGROUP","BUILD\_ID":"33","BASE":"C:\Program Files (x86)\Jenkins","JOB\_NAME":"ELK progress","APPDATA":"C:\WINDOWS\system32\config\systemprofile\AppData\Roaming","NODE\_LABELS":"master","HUDSON\_URL":"[http://10.128.200.4:8080/","windows\_tracing\_logfile":"C:\\BVTBin\\Tests\\installpackage\\csilogfile.log","CommonProgramFiles":"C:\\Program](http://10.128.200.4:8080/%22,%22windows_tracing_logfile%22:%22C:%5C%5CBVTBin%5C%5CTests%5C%5Cinstallpackage%5C%5Ccsilogfile.log%22,%22CommonProgramFiles%22:%22C:%5C%5CProgram) Files (x86)\Common Files","Path":"C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files\Git\cmd;C:\WINDOWS\System32\OpenSSH\;C:\WINDOWS\system32\config\systemprofile\AppData\Local\Microsoft\WindowsApps","OS":"Windows\_NT","windows\_tracing\_flags":"3","COMPUTERNAME":"WIN10X64BASE","EXECUTOR\_NUMBER":"0","HUDSON\_SERVER\_COOKIE":"7a414e699751cd17","JOB\_BASE\_NAME":"ELK progress","PROCESSOR\_REVISION":"4f00","CLASSPATH":"","CommonProgramW6432":"C:\Program Files\Common Files","ComSpec":"C:\WINDOWS\system32\cmd.exe","BUILD\_NUMBER":"33","SystemRoot":"C:\WINDOWS","TEMP":"C:\WINDOWS\TEMP","BUILD\_URL":"[http://10.128.200.4:8080/job/ELK%20progress/33/","USERPROFILE":"C:\\WINDOWS\\system32\\config\\systemprofile","TMP":"C:\\WINDOWS\\TEMP","CommonProgramFiles(x86)":"C:\\Program](http://10.128.200.4:8080/job/ELK%20progress/33/%22,%22USERPROFILE%22:%22C:%5C%5CWINDOWS%5C%5Csystem32%5C%5Cconfig%5C%5Csystemprofile%22,%22TMP%22:%22C:%5C%5CWINDOWS%5C%5CTEMP%22,%22CommonProgramFiles(x86)%22:%22C:%5C%5CProgram) Files (x86)\Common Files","NUMBER\_OF\_PROCESSORS":"4"},"sensitiveBuildVariables":}
> \*\*message":["There is no public Hudson instance configured for this project"]
## \*\*,"source":"jenkins","source\_host":"[http://10.128.200.4:8080/","@buildTimestamp":"2019-01-04T10:21:40-0600","@timestamp":"2019-01-04T10:21:40-0600","@version](http://10.128.200.4:8080/%22,%22@buildTimestamp%22:%222019-01-04T10:21:40-0600%22,%22@timestamp%22:%222019-01-04T10:21:40-0600%22,%22@version)":1}
entry 2
* * *
"data":{"id":"33","result":"SUCCESS","projectName":"ELK progress","fullProjectName":"ELK progress","displayName":"#33","fullDisplayName":"ELK progress #33","url":"job/ELK%20progress/33/","buildHost":"Jenkins","buildLabel":"master","buildNum":33,"buildDuration":0,"rootProjectName":"ELK progress","rootFullProjectName":"ELK progress","rootProjectDisplayName":"#33","rootBuildNum":33,"buildVariables":{"PROCESSOR\_LEVEL":"6","RUN\_DISPLAY\_URL":"[http://10.128.200.4:8080/job/ELK%20progress/33/display/redirect","FP\_NO\_HOST\_CHECK":"NO","HUDSON\_HOME":"C:\\Program](http://10.128.200.4:8080/job/ELK%20progress/33/display/redirect%22,%22FP_NO_HOST_CHECK%22:%22NO%22,%22HUDSON_HOME%22:%22C:%5C%5CProgram) Files (x86)\Jenkins","RUN\_CHANGES\_DISPLAY\_URL":"[http://10.128.200.4:8080/job/ELK%20progress/33/display/redirect?GenuineIntel","JOB\_DISPLAY\_URL":"http://10.128.200.4:8080/job/ELK%20progress/display/redirect","WINSW\_EXECUTABLE":"C:\\Program](http://10.128.200.4:8080/job/ELK%20progress/33/display/redirect?GenuineIntel%22,%22JOB_DISPLAY_URL%22:%22http://10.128.200.4:8080/job/ELK%20progress/display/redirect%22,%22WINSW_EXECUTABLE%22:%22C:%5C%5CProgram) Files (x86)\Jenkins\jenkins.exe","ProgramFiles":"C:\Program Files (x86)","PUBLIC":"C:\Users\Public","windir":"C:\WINDOWS","LOCALAPPDATA":"C:\WINDOWS\system32\config\systemprofile\AppData\Local","USERDOMAIN":"WORKGROUP","BUILD\_ID":"33","BASE":"C:\Program Files (x86)\Jenkins","JOB\_NAME":"ELK progress","APPDATA":"C:\WINDOWS\system32\config\systemprofile\AppData\Roaming","NODE\_LABELS":"master","HUDSON\_URL":"[http://10.128.200.4:8080/","windows\_tracing\_logfile":"C:\\BVTBin\\Tests\\installpackage\\csilogfile.log","CommonProgramFiles":"C:\\Program](http://10.128.200.4:8080/%22,%22windows_tracing_logfile%22:%22C:%5C%5CBVTBin%5C%5CTests%5C%5Cinstallpackage%5C%5Ccsilogfile.log%22,%22CommonProgramFiles%22:%22C:%5C%5CProgram) Files (x86)\Common Files","Path":"C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files\Git\cmd;C:\WINDOWS\System32\OpenSSH\;C:\WINDOWS\system32\config\systemprofile\AppData\Local\Microsoft\WindowsApps","OS":"Windows\_NT","windows\_tracing\_flags":"3","COMPUTERNAME":"WIN10X64BASE","EXECUTOR\_NUMBER":"0","HUDSON\_SERVER\_COOKIE":"7a414e699751cd17","JOB\_BASE\_NAME":"ELK progress","PROCESSOR\_REVISION":"4f00","CLASSPATH":"","CommonProgramW6432":"C:\Program Files\Common Files","ComSpec":"C:\WINDOWS\system32\cmd.exe","BUILD\_NUMBER":"33","SystemRoot":"C:\WINDOWS","TEMP":"C:\WINDOWS\TEMP","BUILD\_URL":"[http://10.128.200.4:8080/job/ELK%20progress/33/","USERPROFILE":"C:\\WINDOWS\\system32\\config\\systemprofile","TMP":"C:\\WINDOWS\\TEMP","CommonProgramFiles(x86)":"C:\\Program](http://10.128.200.4:8080/job/ELK%20progress/33/%22,%22USERPROFILE%22:%22C:%5C%5CWINDOWS%5C%5Csystem32%5C%5Cconfig%5C%5Csystemprofile%22,%22TMP%22:%22C:%5C%5CWINDOWS%5C%5CTEMP%22,%22CommonProgramFiles(x86)%22:%22C:%5C%5CProgram) Files (x86)\Common Files","NUMBER\_OF\_PROCESSORS":"4"},"sensitiveBuildVariables":},"\*
> _message":["Finished: SUCCESS"],_\*
## "source":"jenkins","source\_host":"[http://10.128.200.4:8080/","@buildTimestamp":"2019-01-04T10:21:40-0600","@timestamp":"2019-01-04T10:21:40-0600","@version](http://10.128.200.4:8080/%22,%22@buildTimestamp%22:%222019-01-04T10:21:40-0600%22,%22@timestamp%22:%222019-01-04T10:21:40-0600%22,%22@version)":1}
you will notice that only messge is differnent rest all are metadata which is same. i would like to keep only one meta-data for all the build which has same build number
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [January 7, 2019, 10:25pm UTC](https://discuss.elastic.co/t/removing-metadata-from-logs-in-logstash/162994/2 "2019-01-07T22:25:15Z")
If you want to aggregate all the messages for a build into a single event then an aggregate filter might be able to do it. It is impossible to read you example data. You need to quote it (select the text and click on the \ in the bar above the edit window).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 4, 2019, 10:25pm UTC](https://discuss.elastic.co/t/removing-metadata-from-logs-in-logstash/162994/3 "2019-02-04T22:25:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.