Remove fields with empty value

Elastic Stack Logstash
1

Hello,

I am receiving the data in logstash and I can see that at times, some of the fields do not have any values.

Please see below for :
flowid
compositedetails
causedby

image
image284×549 27.3 KB

I would like to remove these fields if they are empty.

I tried to use ruby. I already had a ruby code in my existing confid and hence I appended the code to the same block.

The code looks like below :

ruby {
code => "
s = event.get('logmessage')
r = s.scan(/Caused [bB]y: ([^\n]+)/)
r = r.flatten
event.set('causedby', r.join('/'))
fid = s.scan(/FlowId: ([^\n]+)/)
fid = fid.flatten
event.set('flowid', fid.join('/'))
cd = s.scan(/ComponentDN: ([^\n]+)/)
cd = cd.flatten
event.set('compositedetails', cd.join('/'))
## REMOVE Fields Starts from HERE
hash = event.to_hash
hash.each do |k,v|
if v == nil
event.remove(k)
end
end
"
}

However, the empty fields are not getting deleted.

Any suggestions here please ??

2

They are not nil, they are empty strings. Try

if v == ""
3

Thank you...I will try to implement this and update back....

4

Hello Badger,

I have made the change as suggested by you but now the logstash does not start with syntax error.

The error while starting logstash is :

[2020-04-30T12:07:08,732][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], "#", "{", "}" at line 73, column 16 (byte 2196) after filter {\n\nif "SOA3Q_SOA1_Access" in [tags] {\ngrok {\nmatch => ["message", "%{IPORHOST:clientip}%{SPACE}%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{WORD:method}%{SPACE}%{NOTSPACE:request}%{SPACE}%{NUMBER:response:int}%{SPACE}%{NUMBER:bytes:int}%{SPACE}%{NUMBER:timetaken:float}"]\n add_field => ["Project", "SOA3Series"]\n add_field => ["Environment", "SOA3Q"]\n add_field => ["Lifecycle Status", "Development"]\n add_field => ["Location", "NC"]\n add_field => ["Log Name", "SOA3Q SOA1 Access Log"]\n add_field => ["Server", "usncx212"]\n}\n\ndate {\n match => [ "timestamp" , "ISO8601" ]\n}\nmutate {\n remove_field => ["timestamp"]\n}\n\nmutate {\n copy => { "request" => "request_tmp" }\n }\n mutate {\n split => { "request_tmp" => "/" }\n add_field => { "partition" => "%{[request_tmp][3]}" }\n add_field => { "composite" => "%{[request_tmp][4]}" }\n add_field => { "component" => "%{[request_tmp][5]}" }\n }\n\nmutate {\n remove_field => ["request_tmp"]\n}\n\n\n}\n\n if "SOA3Q_Domain_Log" in [tags] {\ngrok {\nmatch => ["message", "####<%{DATA:timestamp}> <%{WORD:severity}> <%{DATA:wls_Topic}> <%{DATA:host}> <%{WORD:managedserver}> <%{DATA:field6}> <%{DATA:field7}> <%{DATA:field8}> <%{DATA:field9}> <%{DATA:field10}> <%{DATA:field11}> <(%{DATA:error_code})?> (<%{GREEDYDATA:logmessage})?"]\n\tadd_field => ["Project", "SOA3Series"]\n add_field => ["Environment", "SOA3Q"]\n add_field => ["Lifecycle Status", "Development"]\n add_field => ["Location", "NC"]\n add_field => ["Log Name", "SOA3Q Domain Log"]\n add_field => ["Server", "usncx212"]\n}\nmutate {\n remove_field => ["timestamp"]\n}\n\nruby {\n code => "\n s = event.get('logmessage')\n r = s.scan(/Caused [bB]y: ([^\n]+)/)\n r = r.flatten\n event.set('causedby', r.join('/'))\n fid = s.scan(/FlowId: ([^\n]+)/)\n fid = fid.flatten\n event.set('flowid', fid.join('/'))\n cd = s.scan(/ComponentDN: ([^\n]+)/)\n cd = cd.flatten\n event.set('compositedetails', cd.join('/'))\n \n hash = event.to_hash\n hash.each do |k,v|\n if v == "", :backtrace=>["/elk/logstash/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative'", "/elk/logstash/logstash-core/lib/logstash/compiler.rb:49:in compile_graph'", "/elk/logstash/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2580:in map'", "/elk/logstash/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:161:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in initialize'", "/elk/logstash/logstash-core/lib/logstash/java_pipeline.rb:27:in initialize'", "/elk/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "/elk/logstash/logstash-core/lib/logstash/agent.rb:326:in block in converge_state'"]}
[2020-04-30T12:07:08,883][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-04-30T12:07:13,994][INFO ][logstash.runner ] Logstash shut down.

Below is the logstash config :

if "SOA3Q_Domain_Log" in [tags] {
grok {
match => ["message", "####<%{DATA:timestamp}> <%{WORD:severity}> <%{DATA:wls_Topic}> <%{DATA:host}> <%{WORD:managedserver}> <%{DATA:field6}> <%{DATA:field7}> <%{DATA:field8}> <%{DATA:field9}> <%{DATA:field10}> <%{DATA:field11}> <(%{DATA:error_code})?> (<%{GREEDYDATA:logmessage})?"]
add_field => ["Project", "SOA3Series"]
add_field => ["Environment", "SOA3Q"]
add_field => ["Lifecycle Status", "Development"]
add_field => ["Location", "NC"]
add_field => ["Log Name", "SOA3Q Domain Log"]
add_field => ["Server", "212"]
}
mutate {
remove_field => ["timestamp"]
}

ruby {
code => "
s = event.get('logmessage')
r = s.scan(/Caused [bB]y: ([^\n]+)/)
r = r.flatten
event.set('causedby', r.join('/'))
fid = s.scan(/FlowId: ([^\n]+)/)
fid = fid.flatten
event.set('flowid', fid.join('/'))
cd = s.scan(/ComponentDN: ([^\n]+)/)
cd = cd.flatten
event.set('compositedetails', cd.join('/'))

  hash = event.to_hash
  hash.each do |k,v|
  if v == ""
  event.remove(k)
  end
end

"
}
}

Line 73 corresponds to :

if v == ""

Any changes to be made to the code ???

5

If you are using double quote around the value of the code option then you would need to use single quotes around strings within it. Try

if v == ''
6

Thank you...This is now working...

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.