Null validation in logstash

Elastic Stack Logstash
1

How to check null dates in logstash.
Null validation in logstash.
I added below configuration is it correct
if[datetime]!='null' and [datetime]!='NULL'{

2

No, because you are are now looking for those actual values.

You should be able to use a ruby filter, something like;

        ruby {
                code => "
                        hash = event.to_hash
                        hash.each do |k,v|
                                if v == nil
                                        event.remove(k)
                                end
                        end
                "
        }

You can also do if [datetime] and [datetime] if the fields aren't even being created.

3

To test if a field does not exist, the syntax is :
if ![datetime]

4 Likes
4

in csv i have 4 fields .i wnat to check null validation on one field
for example:
transid,status,datetime

i want add null validation on datetime field.
how to add.

5

can you give csv data example where you have "null" values ?

6

"clientid"=>238, "groupid"=>550, "teamid"=>843, "userid"=>28, "trans_id"=>"1462098183_90927569",
"eventtransid"=>"1462098183_90927569", "serviceid"=>2948, "servicekey"=>"cc01cce5-0aad-11e6-a89d-005056944c2b",
"servicename"=>"DVP GW testing", "crn"=>"2036", "eventid"=>8, "eventtype"=>"MISSED_CALL", "pulsecount"=>1,
"ani"=>"919440324831", "dnis"=>"+914066977906", "answeredon"=>"null", "releasedon"=>"2016-05-01T10:23:03.565Z",
"acceptedon"=>"2016-05-01T10:23:03.501Z", "offeredon"=>"2016-05-01T10:23:03.373Z", "statusid"=>837,
"statusname"=>"voice call summary", "ostype"=>0, "osname"=>"null", "dvpdropstate"=>2, "dvpdropcode"=>2006,

in this answered on is null

1 Like
7

238550843281462098183_909275691462098183_909275692948cc01cce5-0aad-11e6-a89d-005056944c2bDVP GW testing20368MISSED_CALL1919440324831+914066977906null2016-05-01 15:53:03.5652016-05-01 15:53:03.5012016-05-01 15:53:03.373837voice call summary0null22006OthersAnswer/logdata/apps/dvp/callflows/missedcallflow/missedcall.txt102016-05-01 15:53:14.659null00null

8

OK, in your case, date conversion fails, so field is filled with "null".

=> To detect this case, I advice this if :
if [datetime] != 'null' {

No need to check 'NULL'.

2 Likes