# How to handle dictionary from python script in logstash

**URL:** <https://discuss.elastic.co/t/how-to-handle-dictionary-from-python-script-in-logstash/173960>\
**Category:** Logstash\
**Created:** [March 26, 2019, 3:37pm UTC](https://discuss.elastic.co/t/how-to-handle-dictionary-from-python-script-in-logstash/173960 "2019-03-26T15:37:43Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ioannisnikolakis](https://avatars.discourse-cdn.com/v4/letter/i/e79b87/32.png) [@ioannisnikolakis](https://discuss.elastic.co/u/ioannisnikolakis)\
**Post date:** [March 26, 2019, 3:37pm UTC](https://discuss.elastic.co/t/how-to-handle-dictionary-from-python-script-in-logstash/173960/1 "2019-03-26T15:37:43Z")

</div>

Hello elastic and logstash experts,

I am trying out using a python script as input (exec) to a logstash config file. The python script spits out a dictionary like the one depicted below once every 1 second:

```
{'_score': 0.8471497, '_index': 'youtube_v1_2019', '_id': 'Ugzw9hy4KLDexJYgP1J4AaABAg', '_type': 'comment', '_source': {'createdAt': '2019-03-09 11:51:26', 'userPreference': [], 'text': 'Dirollo 4life', 'active': 1, 'username': 'Panos Baxevanos', 'keywords': ['4life', 'dirollo'], 'text_greek': 'Dirollo 4life', 'documentSentiment': 0.0, 'impact': 0.03}}

```

What I want is to transform each of these into a field (with the same name as the key in the dictionary) , and insert it into elasticsearch as a single document.  
How can I achieve this ? I've had problems trying to handle the dictionary, as it is inserted alltogether into a field named "message".

Thank you in advance

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 26, 2019, 3:46pm UTC](https://discuss.elastic.co/t/how-to-handle-dictionary-from-python-script-in-logstash/173960/2 "2019-03-26T15:46:00Z")

</div>

```
mutate { gsub => ["message", "'", '"'] }
json { source => "message" }
```

---

<div class="post-metadata">

**Author:** ![ioannisnikolakis](https://avatars.discourse-cdn.com/v4/letter/i/e79b87/32.png) [@ioannisnikolakis](https://discuss.elastic.co/u/ioannisnikolakis)\
**Post date:** [March 27, 2019, 11:42am UTC](https://discuss.elastic.co/t/how-to-handle-dictionary-from-python-script-in-logstash/173960/3 "2019-03-27T11:42:13Z")

</div>

Thank you very much for your immediate response, it was very helpful.

I have one additional problem though which is related. After applying the json filter, I want to do the same for the nested document "\_source" which I renamed as "re" (short for response), in order to flatten the document. I get the following error:

```
 json - Error parsing json {:source=>"re", :raw=>{"createdAt"=>"2019-03-09 11:51:26", 
"documentSentiment"=>#<BigDecimal:5764d11b,'0.0',1(4)>, "userPreference"=>[], 
"username"=>"Panos Baxevanos", "keywords"=>["4life", "dirollo"], "text_greek"=>"Dirollo 4life", 
"text"=>"Dirollo 4life", "impact"=>#<BigDecimal:63b68a73,'0.3E-1',1(4)>, "active"=>1}, 
:exception=>java.lang.ClassCastException: org.jruby.RubyHash cannot be cast to 
org.jruby.RubyIO}

```

This should be a valid json, but it isn't recognised as one I suppose. Any thoughts on this ?

Again thank you in advance !

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 27, 2019, 1:23pm UTC](https://discuss.elastic.co/t/how-to-handle-dictionary-from-python-script-in-logstash/173960/4 "2019-03-27T13:23:06Z")

</div>

The json filter will parse a string that is JSON, including nested objects. When you parse "message", "\_source" is no longer a string, it is an object, so you cannot pass it to a json filter.

But you can refer to the nested fields. For example...

```
mutate { add_field => { "secondKeyword" => "%{[_source][keywords][1]}" } }
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 24, 2019, 1:30pm UTC](https://discuss.elastic.co/t/how-to-handle-dictionary-from-python-script-in-logstash/173960/5 "2019-04-24T13:30:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.