# Logstash assign key value pair into each variables **URL:** \ **Category:** Logstash\ **Created:** [May 21, 2017, 5:09pm UTC](https://discuss.elastic.co/t/logstash-assign-key-value-pair-into-each-variables/86583 "2017-05-21T17:09:51Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![vallikkv](https://avatars.discourse-cdn.com/v4/letter/v/e480ec/32.png) [@vallikkv](https://discuss.elastic.co/u/vallikkv)\ **Post date:** [May 21, 2017, 5:09pm UTC](https://discuss.elastic.co/t/logstash-assign-key-value-pair-into-each-variables/86583/1 "2017-05-21T17:09:51Z")
Hello All, I'm 3 day new to log stash and during this learning process, I tried to filter the unstructured log which contains key & value pairs and want to assign each values to variables. **Unstructured log:** 017-04-04 16:01:16,726 - test.test. - DEBUG - notify\_event=[u'[{"Key1":"Value1","Key2":Value2,"Key3": Value3,"}]'] **Configuration file:** input { file { path =\> "/home/vallikkv/ELK/log/messages" type =\> "messages" } } filter { grok { match =\> { "message" =\> "%{TIMESTAMP\_ISO8601} - test.test. - %{LOGLEVEL} - notify\_event=%{GREEDYDATA:msg}" } } json{ source =\> msg } } output { stdout{} } **Questions:** 1. How to get the values of each keys of the GREEDYDATA or msg and save it in a file? 2. How can I pass only these key & value pairs to elasticsearch? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 21, 2017, 6:16pm UTC](https://discuss.elastic.co/t/logstash-assign-key-value-pair-into-each-variables/86583/2 "2017-05-21T18:16:08Z")
It looks like the JSON part of the log is malformed. If it was valid JSON, your configuration probably should work as long as you modify the GREEDYMESSAGE to just capture `{"Key1":"Value1","Key2":Value2,"Key3":"Value3"}`. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 22, 2017, 6:18am UTC](https://discuss.elastic.co/t/logstash-assign-key-value-pair-into-each-variables/86583/3 "2017-05-22T06:18:23Z")
Is "Value2" and "Value3" really not double-quoted like "Value1" is? ---
**Author:** ![vallikkv](https://avatars.discourse-cdn.com/v4/letter/v/e480ec/32.png) [@vallikkv](https://discuss.elastic.co/u/vallikkv)\ **Post date:** [May 22, 2017, 6:31am UTC](https://discuss.elastic.co/t/logstash-assign-key-value-pair-into-each-variables/86583/4 "2017-05-22T06:31:44Z")
Yes, Double quoted contains string values and no quotes for numbers/integers. Eg: {"Name": "Vallinayagam", "Age": 30} ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 22, 2017, 7:24am UTC](https://discuss.elastic.co/t/logstash-assign-key-value-pair-into-each-variables/86583/5 "2017-05-22T07:24:57Z")
Given that the example is invalid JSON and looks made up, it might be better to show a real event. ---
**Author:** ![vallikkv](https://avatars.discourse-cdn.com/v4/letter/v/e480ec/32.png) [@vallikkv](https://discuss.elastic.co/u/vallikkv)\ **Post date:** [May 22, 2017, 8:13am UTC](https://discuss.elastic.co/t/logstash-assign-key-value-pair-into-each-variables/86583/6 "2017-05-22T08:13:01Z")
**Real event:** 2017-04-13 11:50:58,272 - test.TEST. - DEBUG - notify\_event=[u'[{"Description":"Card expid","Location":"Main Door","LogTime":"2017-04-13T11:50:58","OccurenceTime":"2017-04-13T11:50:57","Token":"123456","Type":4,"OperatorDetail":{"FirstName":"Simon","LastName":"Rampling"}}]'] ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 22, 2017, 12:29pm UTC](https://discuss.elastic.co/t/logstash-assign-key-value-pair-into-each-variables/86583/7 "2017-05-22T12:29:53Z")
Okay, so what follows "notify\_event=" looks like a stringified Python array containing a single string value. You can use grok to only capture what's between `notify_event=[u'` and `']`, then feed the extracted JSON string to a json filter. ---
**Author:** ![vallikkv](https://avatars.discourse-cdn.com/v4/letter/v/e480ec/32.png) [@vallikkv](https://discuss.elastic.co/u/vallikkv)\ **Post date:** [May 22, 2017, 1:08pm UTC](https://discuss.elastic.co/t/logstash-assign-key-value-pair-into-each-variables/86583/8 "2017-05-22T13:08:40Z")
Thanks for pointing out, I changed the grok filter as below filter { grok { match =\> { "message" =\> "%{TIMESTAMP\_ISO8601} - test.TEST. - %{LOGLEVEL} - notify\_event=[%{WORD}'[%{GREEDYDATA:msg}]']" } } json{ source =\> msg } } and it worked. But I've some basic questions w.r.t display in kibana. **Configured output as below:** output { elasticsearch { hosts =\> ["localhost:9200"] } stdout{codec =\> "rubydebug"} } 1. How to view these logs in kibana & search in elasticsearch? 2. How these outputs as redirected to elasticsearch? Is it realtime data or how to save it in a file? Any suggestions? Thanks. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 23, 2017, 7:21am UTC](https://discuss.elastic.co/t/logstash-assign-key-value-pair-into-each-variables/86583/9 "2017-05-23T07:21:05Z")
> How to view these logs in kibana & search in elasticsearch? Go to Kibana, configure an index pattern that matches your index(es), and you're done. With your configuration Logstash is going to write to indexes named logstash-YYYY.MM.DD, where YYYY.MM.DD is the date when an event occurred. > How these outputs as redirected to elasticsearch? Is it realtime data or how to save it in a file? Logstash sends the data to Elasticsearch continuously. If things are working okay events written to log files should be searchable in Kibana within a few seconds. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 20, 2017, 7:21am UTC](https://discuss.elastic.co/t/logstash-assign-key-value-pair-into-each-variables/86583/10 "2017-06-20T07:21:41Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.