[Solved]How to convert a json format string to json object for only one field of a full log text in ruby filter?

#1

I use filebeat to collcet log.Then push log to kafka ,then use logstash to parse every log record.then push to another kafka again.then use another logstash push the log to elasticsearch.

And the log is splited by '\t'.one field('respone_body') is json format string like below

{"ret":0,"msg":"","data":[]}

how can i convert it to json in ruby filter?
because my elasticsearch defined the response_body as a mapping with object type and dynamic true.

below is my logstash ruby filter.

ruby {
          init => "@kname = ['datetime','level','logger','thread','version','respose_body']"
          code => 'event.append(Hash[@kname.zip(event["message"].split("\t"))])'
          remove_field => ['input_type','beat','host','message','offset','kafka'] 
    }

and this is log, it is a full string with 5 tablespace.

2019-05-09-15:00:04.258	ACCESS	ad_service\inc\Router		333 	0.1	{"ret":0,"msg":"","data":[]}

I have tried below solution,But still not work.
ruby filter code.

event["json_respone_body"] = event["respone_body"].to_json

Logstatsh error information:

{:timestamp=>"2019-05-10T18:08:19.938000+0800", :message=>"Ruby exception occurred: uninitialized constant LogStash::Filters::Ruby::JSON", :level=>:error}
#2

Why not use a json filter?

#4

because just one field is json format.But the full log record is still recogized as string.

#5

somebody can give some advices? thanks a lot.

#6

Use a json filter. You tell the filter which field to parse. It does not have to be message, it can be anything.

#7

but the other fields have to be deal with as string.And I hide some ruby code for the other fields.
you mean filter deat twice,the first time use ruby code,and the second time use json filter?
how can I configure it? like below?

 json {
        source => "%{respone_body}"
        add_field => {"json_respone_body" => "%{respone_body}"}
    }
 json {
        source => event["respone_body"]
        target => event["json_respone_body"]
    }

the two I tried,looks like still not working...

#8

use this works

 json {
        source => "respone_body"
        target => "json_respone_body"
    }