Extract Data from message to display each field as a column in kibana

If your input is valid JSON, or even close to it, then a json filter is most likely better than grok. Now, for the sample data you showed in the first post, you need to configure the input so that rubydebug shows the entire JSON object in a single event. Like this...

"message" => "{ \"_id\" : ObjectId(\"5a21e54533015\"), \"clientCode\" : \"demo\", \"conversationId\" : \"d6416ec0--930f-da7aa79f3215\", \"employeeId\" : \"45\", \"entities\" : [ { \"entity\" : \"status\", \"location\" : [ NumberInt(\"0\"), NumberInt(\"2\") ], \"value\" : \"ok\", \"confidence\" : NumberInt(\"1\") } ], \"input\" : { \"feedback\" : { \"feedbackSubject\" : \"my feedbac\", \"feedbackText\" : \"feedback\\nthis is good\\nI love this\", \"feedbackCategory\" : \"\", \"conversationId\" : \"d6416ec0-2f9a-42fb-930f-da7aa79f3215\", \"conversationText\" : \"(HI) [Greetings, human.]\", \"conversationNodeName\" : \"root\" } }, \"intents\" : [ { \"intent\" : \"feedbackresponse\", \"confidence\" : NumberInt(\"1\") } ], \"locale\" : \"en-ca\" }",

as opposed to what you have now, which is

"message" => "\t"input" : {",

If you want to consume the entire file as a single event then you will need to use a multiline code. You can use the trick described here, of appending a line that is known not to occur in the input. Some people will recommend using auto_flush_interval, but personally I think that is an ugly hack.