# Remove fields with empty value
**URL:** https://discuss.elastic.co/t/remove-fields-with-empty-value/230470
**Category:** Logstash
**Created:** [April 30, 2020, 6:26am UTC](https://discuss.elastic.co/t/remove-fields-with-empty-value/230470 "2020-04-30T06:26:31Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@vikramaddagulla](https://discuss.elastic.co/u/vikramaddagulla)
#### Post date: [April 30, 2020, 6:26am UTC](https://discuss.elastic.co/t/remove-fields-with-empty-value/230470/1 "2020-04-30T06:26:31Z")
Hello,
I am receiving the data in logstash and I can see that at times, some of the fields do not have any values.
Please see below for :
flowid
compositedetails
causedby
I would like to remove these fields if they are empty.
I tried to use ruby. I already had a ruby code in my existing confid and hence I appended the code to the same block.
The code looks like below :
ruby {
code =\> "
s = event.get('logmessage')
r = s.scan(/Caused [bB]y: ([^\n]+)/)
r = r.flatten
event.set('causedby', r.join('/'))
fid = s.scan(/FlowId: ([^\n]+)/)
fid = fid.flatten
event.set('flowid', fid.join('/'))
cd = s.scan(/ComponentDN: ([^\n]+)/)
cd = cd.flatten
event.set('compositedetails', cd.join('/'))
## REMOVE Fields Starts from HERE
hash = event.to\_hash
hash.each do |k,v|
if v == nil
event.remove(k)
end
end
"
}
However, the empty fields are not getting deleted.
Any suggestions here please ??
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [April 30, 2020, 2:57pm UTC](https://discuss.elastic.co/t/remove-fields-with-empty-value/230470/2 "2020-04-30T14:57:24Z")
> [@vikramaddagulla](#):
>
> if v == nil
They are not nil, they are empty strings. Try
```
if v == ""
```
---
### Author:  [@vikramaddagulla](https://discuss.elastic.co/u/vikramaddagulla)
#### Post date: [April 30, 2020, 4:02pm UTC](https://discuss.elastic.co/t/remove-fields-with-empty-value/230470/3 "2020-04-30T16:02:53Z")
Thank you...I will try to implement this and update back....
---
### Author:  [@vikramaddagulla](https://discuss.elastic.co/u/vikramaddagulla)
#### Post date: [April 30, 2020, 4:11pm UTC](https://discuss.elastic.co/t/remove-fields-with-empty-value/230470/4 "2020-04-30T16:11:54Z")
Hello Badger,
I have made the change as suggested by you but now the logstash does not start with syntax error.
The error while starting logstash is :
[2020-04-30T12:07:08,732][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of [\t\r\n], "#", "{", "}" at line 73, column 16 (byte 2196) after filter {\n\nif "SOA3Q\_SOA1\_Access" in [tags] {\ngrok {\nmatch =\> ["message", "%{IPORHOST:clientip}%{SPACE}%{TIMESTAMP\_ISO8601:timestamp}%{SPACE}%{WORD:method}%{SPACE}%{NOTSPACE:request}%{SPACE}%{NUMBER:response:int}%{SPACE}%{NUMBER:bytes:int}%{SPACE}%{NUMBER:timetaken:float}"]\n add\_field =\> ["Project", "SOA3Series"]\n add\_field =\> ["Environment", "SOA3Q"]\n add\_field =\> ["Lifecycle Status", "Development"]\n add\_field =\> ["Location", "NC"]\n add\_field =\> ["Log Name", "SOA3Q SOA1 Access Log"]\n add\_field =\> ["Server", "usncx212"]\n}\n\ndate {\n match =\> ["timestamp" , "ISO8601"]\n}\nmutate {\n remove\_field =\> ["timestamp"]\n}\n\nmutate {\n copy =\> { "request" =\> "request\_tmp" }\n }\n mutate {\n split =\> { "request\_tmp" =\> "/" }\n add\_field =\> { "partition" =\> "%{[request\_tmp][3]}" }\n add\_field =\> { "composite" =\> "%{[request\_tmp][4]}" }\n add\_field =\> { "component" =\> "%{[request\_tmp][5]}" }\n }\n\nmutate {\n remove\_field =\> ["request\_tmp"]\n}\n\n\n}\n\n if "SOA3Q\_Domain\_Log" in [tags] {\ngrok {\nmatch =\> ["message", "####\<%{DATA:timestamp}\> \<%{WORD:severity}\> \<%{DATA:wls\_Topic}\> \<%{DATA:host}\> \<%{WORD:managedserver}\> \<%{DATA:field6}\> \<%{DATA:field7}\> \<%{DATA:field8}\> \<%{DATA:field9}\> \<%{DATA:field10}\> \<%{DATA:field11}\> \<(%{DATA:error\_code})?\> (\<%{GREEDYDATA:logmessage})?"]\n\tadd\_field =\> ["Project", "SOA3Series"]\n add\_field =\> ["Environment", "SOA3Q"]\n add\_field =\> ["Lifecycle Status", "Development"]\n add\_field =\> ["Location", "NC"]\n add\_field =\> ["Log Name", "SOA3Q Domain Log"]\n add\_field =\> ["Server", "usncx212"]\n}\nmutate {\n remove\_field =\> ["timestamp"]\n}\n\nruby {\n code =\> "\n s = event.get('logmessage')\n r = s.scan(/Caused [bB]y: ([^\n]+)/)\n r = r.flatten\n event.set('causedby', r.join('/'))\n fid = s.scan(/FlowId: ([^\n]+)/)\n fid = fid.flatten\n event.set('flowid', fid.join('/'))\n cd = s.scan(/ComponentDN: ([^\n]+)/)\n cd = cd.flatten\n event.set('compositedetails', cd.join('/'))\n \n hash = event.to\_hash\n hash.each do |k,v|\n if v == "", :backtrace=\>["/elk/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/elk/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile\_graph'", "/elk/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2580:in `map'", "/elk/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:161:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/elk/logstash/logstash-core/lib/logstash/java_pipeline.rb:27:in `initialize'", "/elk/logstash/logstash-core/lib/logstash/pipeline\_action/create.rb:36:in `execute'", "/elk/logstash/logstash-core/lib/logstash/agent.rb:326:in `block in converge\_state'"]}
[2020-04-30T12:07:08,883][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}
[2020-04-30T12:07:13,994][INFO][logstash.runner] Logstash shut down.
Below is the logstash config :
if "SOA3Q\_Domain\_Log" in [tags] {
grok {
match =\> ["message", "####\<%{DATA:timestamp}\> \<%{WORD:severity}\> \<%{DATA:wls\_Topic}\> \<%{DATA:host}\> \<%{WORD:managedserver}\> \<%{DATA:field6}\> \<%{DATA:field7}\> \<%{DATA:field8}\> \<%{DATA:field9}\> \<%{DATA:field10}\> \<%{DATA:field11}\> \<(%{DATA:error\_code})?\> (\<%{GREEDYDATA:logmessage})?"]
add\_field =\> ["Project", "SOA3Series"]
add\_field =\> ["Environment", "SOA3Q"]
add\_field =\> ["Lifecycle Status", "Development"]
add\_field =\> ["Location", "NC"]
add\_field =\> ["Log Name", "SOA3Q Domain Log"]
add\_field =\> ["Server", "212"]
}
mutate {
remove\_field =\> ["timestamp"]
}
ruby {
code =\> "
s = event.get('logmessage')
r = s.scan(/Caused [bB]y: ([^\n]+)/)
r = r.flatten
event.set('causedby', r.join('/'))
fid = s.scan(/FlowId: ([^\n]+)/)
fid = fid.flatten
event.set('flowid', fid.join('/'))
cd = s.scan(/ComponentDN: ([^\n]+)/)
cd = cd.flatten
event.set('compositedetails', cd.join('/'))
```
hash = event.to_hash
hash.each do |k,v|
if v == ""
event.remove(k)
end
end
```
"
}
}
Line 73 corresponds to :
if v == ""
Any changes to be made to the code ???
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [April 30, 2020, 5:02pm UTC](https://discuss.elastic.co/t/remove-fields-with-empty-value/230470/5 "2020-04-30T17:02:25Z")
> [@vikramaddagulla](#):
>
> ruby {\n code =\> "
If you are using double quote around the value of the code option then you would need to use single quotes around strings within it. Try
```
if v == ''
```
---
### Author:  [@vikramaddagulla](https://discuss.elastic.co/u/vikramaddagulla)
#### Post date: [May 1, 2020, 11:41am UTC](https://discuss.elastic.co/t/remove-fields-with-empty-value/230470/6 "2020-05-01T11:41:09Z")
Thank you...This is now working...
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 29, 2020, 11:42am UTC](https://discuss.elastic.co/t/remove-fields-with-empty-value/230470/7 "2020-05-29T11:42:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.