Event.set and nested filed

[arp220 (david)]

Hi.
I have this config:

input {
  file {
     path => "/opt/logs/sb_log.json"
     sincedb_path => "/etc/logstash/.sincedb_sblog"
     start_position => "beginning"
   }
}
filter{
  json {
        source => "message"
  }
  json {
    source => "payload"
    target => "parsepaylaod"
  }
  json {
    source => "headers"
    target => "parseheaders"
  }
  mutate {
    convert => ["parentId","integer"]
    convert => ["currentNano","integer"]
    convert => ["createTimeMillis","integer"]
    convert => ["elapse","float"]
  }

  ruby {
      code => "event.set('diff',(event.get(['_source']['parseheaders']['kafka_receivedTimestamp']) - event.get('createTimeMillis')))" 
   }
 mutate {
        remove_field => ["@version" ]
  }

if "_jsonparsefailure"  in [tags] { drop {} }
}

output {
     elasticsearch {
        hosts => ["http://192.168.0.39:9200"]
        user => "user"
        password => "pass"
        index => "logstash_index_beta"
    }
}

I have two questions, my syntax in ruby is correct? Do I cat get parseheaders.kafka_receivedTimestamp value this way?

code => "event.set('diff',(event.get(['_source']['parseheaders']['kafka_receivedTimestamp']) - event.get('createTimeMillis')))" 

another this parseheaders.kafka_receivedTimestamp and createTimeMillis is integer type but I get this error

Sep 23 20:22:18 app1 logstash[31508]: [2020-09-23T20:22:18,571][ERROR][logstash.filters.ruby    ][main][c86aa057dfd82fb3bc82c6597913f6166b84156506f117224d0d3540263942d5] Ruby exception occurred: no implicit conversion of String into Integer
Sep 23 20:22:18 app1 logstash[31508]: [2020-09-23T20:22:18,573][ERROR][logstash.filters.ruby    ][main][c86aa057dfd82fb3bc82c6597913f6166b84156506f117224d0d3540263942d5] Ruby exception occurred: no implicit conversion of String into Integer
Sep 23 20:22:18 app1 logstash[31508]: [2020-09-23T20:22:18,574][ERROR][logstash.filters.ruby    ][main][c86aa057dfd82fb3bc82c6597913f6166b84156506f117224d0d3540263942d5] Ruby exception occurred: no implicit conversion of String into Integer

thanks a lot for your help.

[Badger]

@arp220

  ruby {
      code => "event.set('diff',(event.get(['_source']['parseheaders']['kafka_receivedTimestamp']) - event.get('createTimeMillis')))" 
   }

_source is an elasticsearch artifact used to show you the document that was indexed. It does not generally exist in logstash. You probably want

ruby {
    code => "event.set('diff',(event.get('[parseheaders][kafka_receivedTimestamp]').to_i - event.get('createTimeMillis').to_i))" 
}

If those fields are strings in logstash you need to explicitly call .to_i to convert them to integers (or use mutate+convert before the ruby filter) so that the - operator will work.

[arp220 (david)]

thanks, @Badger, this problem solved, but why in kibana I can't create visualize for this filed diff?
How do I save this field so that it can be used in visualize Kibana?

thanks.

[Badger]

Did you refresh the index pattern?

[arp220 (david)]

I love you @Badger