How to handle dictionary from python script in logstash

[ioannisnikolakis (Giannis Nikolakis)]

Hello elastic and logstash experts,

I am trying out using a python script as input (exec) to a logstash config file. The python script spits out a dictionary like the one depicted below once every 1 second:

{'_score': 0.8471497, '_index': 'youtube_v1_2019', '_id': 'Ugzw9hy4KLDexJYgP1J4AaABAg', '_type': 'comment', '_source': {'createdAt': '2019-03-09 11:51:26', 'userPreference': [], 'text': 'Dirollo 4life', 'active': 1, 'username': 'Panos Baxevanos', 'keywords': ['4life', 'dirollo'], 'text_greek': 'Dirollo 4life', 'documentSentiment': 0.0, 'impact': 0.03}}

What I want is to transform each of these into a field (with the same name as the key in the dictionary) , and insert it into elasticsearch as a single document.
How can I achieve this ? I've had problems trying to handle the dictionary, as it is inserted alltogether into a field named "message".

Thank you in advance

[Badger]

mutate { gsub => [ "message", "'", '"' ] }
json { source => "message" }

[ioannisnikolakis (Giannis Nikolakis)]

Thank you very much for your immediate response, it was very helpful.

I have one additional problem though which is related. After applying the json filter, I want to do the same for the nested document "_source" which I renamed as "re" (short for response), in order to flatten the document. I get the following error:

 json - Error parsing json {:source=>"re", :raw=>{"createdAt"=>"2019-03-09 11:51:26", 
"documentSentiment"=>#<BigDecimal:5764d11b,'0.0',1(4)>, "userPreference"=>[], 
"username"=>"Panos Baxevanos", "keywords"=>["4life", "dirollo"], "text_greek"=>"Dirollo 4life", 
"text"=>"Dirollo 4life", "impact"=>#<BigDecimal:63b68a73,'0.3E-1',1(4)>, "active"=>1}, 
:exception=>java.lang.ClassCastException: org.jruby.RubyHash cannot be cast to 
org.jruby.RubyIO}

This should be a valid json, but it isn't recognised as one I suppose. Any thoughts on this ?

Again thank you in advance !

[Badger]

The json filter will parse a string that is JSON, including nested objects. When you parse "message", "_source" is no longer a string, it is an object, so you cannot pass it to a json filter.

But you can refer to the nested fields. For example...

mutate { add_field => { "secondKeyword" => "%{[_source][keywords][1]}" } }