# How do we convert from a hash to json before event reaches the grok filter
**URL:** https://discuss.elastic.co/t/how-do-we-convert-from-a-hash-to-json-before-event-reaches-the-grok-filter/29445
**Category:** Logstash
**Created:** [September 16, 2015, 9:18pm UTC](https://discuss.elastic.co/t/how-do-we-convert-from-a-hash-to-json-before-event-reaches-the-grok-filter/29445 "2015-09-16T21:18:55Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@tmo\_bgc](https://discuss.elastic.co/u/tmo_bgc)
#### Post date: [September 16, 2015, 9:18pm UTC](https://discuss.elastic.co/t/how-do-we-convert-from-a-hash-to-json-before-event-reaches-the-grok-filter/29445/1 "2015-09-16T21:18:55Z")
How do we convert from a hash to json before event reaches the grok filter. The JMX input plugin produces a Hash, but the Grok filter expects JSON.
input {
jmx {
path =\> "c:/LogStash/jmxconf/"
polling\_frequency =\> 15
type =\> "jmx"
nb\_thread =\> 4
codec =\> json
}
}
filter {
grok {
patterns\_dir =\> "C:/LogStash/bak/patterns"
match =\> { "event" =\> "Memory.HeapMemoryUsage.committed", "metric\_value\_number"=\>%{NUMBER:committed}" }
}
}
output {
stdout { codec =\> rubydebug }
}
many thanks,
Tony
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [September 17, 2015, 3:40am UTC](https://discuss.elastic.co/t/how-do-we-convert-from-a-hash-to-json-before-event-reaches-the-grok-filter/29445/2 "2015-09-17T03:40:00Z")
It looks like you're trying to parse JSON with your grok filter. Use a [json filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html) instead. Although I don't get why that's necessary since the jmx input has `codec => json` so it should already be decoding the JSON payload. Could you disable your grok filter and just show us the output from the stdout output so that we can see exactly what we're dealing with?
---
### Author:  [@tmo\_bgc](https://discuss.elastic.co/u/tmo_bgc)
#### Post date: [September 18, 2015, 4:35pm UTC](https://discuss.elastic.co/t/how-do-we-convert-from-a-hash-to-json-before-event-reaches-the-grok-filter/29445/3 "2015-09-18T16:35:31Z")
> [@tmo\_bgc](#):
>
> output { stdout { codec =\> rubydebug }}
Thanks for looking into this. After looking at the JMX plugin code, it doesn't seem like the code does anything with "codec =\> json" setting. On a side note the output stdout code does. I disabled the Grok filter. and the output looks like this:
{
"@version" =\> "1",
"@timestamp" =\> "2015-09-18T16:32:48.191Z",
"host" =\> "[usbgc-lab-vd1.ny.cantor.com](http://usbgc-lab-vd1.ny.cantor.com)",
"path" =\> "c:/LogStash/jmxconf/",
"type" =\> "jmx",
"metric\_path" =\> "usbgc-lab-vd1.ny.cantor.com\_9119.Memory.Verbose\_bo
ol",
"metric\_value\_number" =\> 0
}
{
"@version" =\> "1",
"@timestamp" =\> "2015-09-18T16:32:48.228Z",
"host" =\> "[usbgc-lab-vd1.ny.cantor.com](http://usbgc-lab-vd1.ny.cantor.com)",
"path" =\> "c:/LogStash/jmxconf/",
"type" =\> "jmx",
"metric\_path" =\> "usbgc-lab-vd1.ny.cantor.com\_9119.Memory.HeapMemory
Usage.committed",
"metric\_value\_number" =\> 192937984
}
{
"@version" =\> "1",
"@timestamp" =\> "2015-09-18T16:32:48.243Z",
"host" =\> "[usbgc-lab-vd1.ny.cantor.com](http://usbgc-lab-vd1.ny.cantor.com)",
"path" =\> "c:/LogStash/jmxconf/",
"type" =\> "jmx",
"metric\_path" =\> "usbgc-lab-vd1.ny.cantor.com\_9119.Memory.HeapMemory
Usage.init",
"metric\_value\_number" =\> 162068032
}
{
"@version" =\> "1",
"@timestamp" =\> "2015-09-18T16:32:48.251Z",
"host" =\> "[usbgc-lab-vd1.ny.cantor.com](http://usbgc-lab-vd1.ny.cantor.com)",
"path" =\> "c:/LogStash/jmxconf/",
"type" =\> "jmx",
"metric\_path" =\> "usbgc-lab-vd1.ny.cantor.com\_9119.Memory.HeapMemory
Usage.max",
"metric\_value\_number" =\> 7635730432
}
{
"@version" =\> "1",
"@timestamp" =\> "2015-09-18T16:32:48.258Z",
"host" =\> "[usbgc-lab-vd1.ny.cantor.com](http://usbgc-lab-vd1.ny.cantor.com)",
"path" =\> "c:/LogStash/jmxconf/",
"type" =\> "jmx",
"metric\_path" =\> "usbgc-lab-vd1.ny.cantor.com\_9119.Memory.HeapMemory
Usage.used",
"metric\_value\_number" =\> 141016808
}
{
"@version" =\> "1",
"@timestamp" =\> "2015-09-18T16:32:48.273Z",
"host" =\> "[usbgc-lab-vd1.ny.cantor.com](http://usbgc-lab-vd1.ny.cantor.com)",
"path" =\> "c:/LogStash/jmxconf/",
"type" =\> "jmx",
"metric\_path" =\> "usbgc-lab-vd1.ny.cantor.com\_9119.Memory.NonHeapMem
oryUsage.committed",
"metric\_value\_number" =\> 109641728
}
{
"@version" =\> "1",
"@timestamp" =\> "2015-09-18T16:32:48.280Z",
"host" =\> "[usbgc-lab-vd1.ny.cantor.com](http://usbgc-lab-vd1.ny.cantor.com)",
"path" =\> "c:/LogStash/jmxconf/",
"type" =\> "jmx",
"metric\_path" =\> "usbgc-lab-vd1.ny.cantor.com\_9119.Memory.NonHeapMem
oryUsage.init",
"metric\_value\_number" =\> 24576000
}
{
"@version" =\> "1",
"@timestamp" =\> "2015-09-18T16:32:48.287Z",
"host" =\> "[usbgc-lab-vd1.ny.cantor.com](http://usbgc-lab-vd1.ny.cantor.com)",
"path" =\> "c:/LogStash/jmxconf/",
"type" =\> "jmx",
"metric\_path" =\> "usbgc-lab-vd1.ny.cantor.com\_9119.Memory.NonHeapMem
oryUsage.max",
"metric\_value\_number" =\> 587202560
}
{
"@version" =\> "1",
"@timestamp" =\> "2015-09-18T16:32:48.319Z",
"host" =\> "[usbgc-lab-vd1.ny.cantor.com](http://usbgc-lab-vd1.ny.cantor.com)",
"path" =\> "c:/LogStash/jmxconf/",
"type" =\> "jmx",
"metric\_path" =\> "usbgc-lab-vd1.ny.cantor.com\_9119.Memory.NonHeapMem
oryUsage.used",
"metric\_value\_number" =\> 60404040
}
{
"@version" =\> "1",
"@timestamp" =\> "2015-09-18T16:32:48.323Z",
"host" =\> "[usbgc-lab-vd1.ny.cantor.com](http://usbgc-lab-vd1.ny.cantor.com)",
"path" =\> "c:/LogStash/jmxconf/",
"type" =\> "jmx",
"metric\_path" =\> "usbgc-lab-vd1.ny.cantor.com\_9119.Memory.ObjectPend
ingFinalizationCount",
"metric\_value\_number" =\> 0
}
{
"@version" =\> "1",
"@timestamp" =\> "2015-09-18T16:32:48.335Z",
"host" =\> "[usbgc-lab-vd1.ny.cantor.com](http://usbgc-lab-vd1.ny.cantor.com)",
"path" =\> "c:/LogStash/jmxconf/",
"type" =\> "jmx",
"metric\_path" =\> "usbgc-lab-vd1.ny.cantor.com\_9119.Memory.ObjectName
",
"metric\_value\_string" =\> "java.lang:type=Memory"
}
---
### Author:  [@tmo\_bgc](https://discuss.elastic.co/u/tmo_bgc)
#### Post date: [September 18, 2015, 4:53pm UTC](https://discuss.elastic.co/t/how-do-we-convert-from-a-hash-to-json-before-event-reaches-the-grok-filter/29445/4 "2015-09-18T16:53:48Z")
Am I correct when I say the output just listed is a Hash? Sorry, i'm new to Logstash and JSON in general. I want to convert from Hash to JSON.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [September 19, 2015, 5:08pm UTC](https://discuss.elastic.co/t/how-do-we-convert-from-a-hash-to-json-before-event-reaches-the-grok-filter/29445/5 "2015-09-19T17:08:51Z")
> {
> "@version" =\> "1",
> "@timestamp" =\> "2015-09-18T16:32:48.280Z",
> "host" =\> "[usbgc-lab-vd1.ny.cantor.com](http://usbgc-lab-vd1.ny.cantor.com)",
> "path" =\> "c:/LogStash/jmxconf/",
> "type" =\> "jmx",
> "metric\_path" =\> "usbgc-lab-vd1.ny.cantor.com\_9119.Memory.NonHeapMem
> oryUsage.init",
> "metric\_value\_number" =\> 24576000
> }
Depending on what kind of output you're interested in it's arguable that there's nothing here to parse. You already have the metric name and its value in discrete fields.
> Am I correct when I say the output just listed is a Hash? Sorry, i'm new to Logstash and JSON in general. I want to convert from Hash to JSON.
The internal representation of a message inside Logstash might be a hash, but that's not something you should be concerned with. What's relevant is that it's a set of key/value pairs. Each message is converted upon output to e.g. JSON or, in the case above, a Ruby-specific format that's hardly useful for anything except debugging.
The jmx input produces key/value pairs from the start so you don't need to do any parsing. A grok filter is used to extract values from a flat string into a bunch of key/value pairs, but that's again nothing you need to do.
---
### Author:  [@tmo\_bgc](https://discuss.elastic.co/u/tmo_bgc)
#### Post date: [September 21, 2015, 6:55pm UTC](https://discuss.elastic.co/t/how-do-we-convert-from-a-hash-to-json-before-event-reaches-the-grok-filter/29445/6 "2015-09-21T18:55:56Z")
Thanks for the information. With it and some probing i'm able to see what you say as well as produce an example to do what I need. However I do have a new question: In the output that I posted earlier displays multiple events, meaning each are individual separate events. ie:
{
"@version" =\> "1",
"@timestamp" =\> "2015-09-18T16:32:48.228Z",
"host" =\> "[usbgc-lab-vd1.ny.cantor.com](http://usbgc-lab-vd1.ny.cantor.com)",
"path" =\> "c:/LogStash/jmxconf/",
"type" =\> "jmx",
"metric\_path" =\> "usbgc-lab-vd1.ny.cantor.com\_9119.Memory.HeapMemory
Usage.committed",
"metric\_value\_number" =\> 192937984
}
{
"@version" =\> "1",
"@timestamp" =\> "2015-09-18T16:32:48.243Z",
"host" =\> "[usbgc-lab-vd1.ny.cantor.com](http://usbgc-lab-vd1.ny.cantor.com)",
"path" =\> "c:/LogStash/jmxconf/",
"type" =\> "jmx",
"metric\_path" =\> "usbgc-lab-vd1.ny.cantor.com\_9119.Memory.HeapMemory
Usage.init",
"metric\_value\_number" =\> 162068032
}
...
...
...
How do I merge them into a single event? As a result I can get committed, and init JMX memory information as one event and post it to Elastic Search as one event. A little more insight into what i'm trying to do. I have Kibana connected to elastic search and want to see that single event as a datapoint on one of my graphs.
many thanks,
Tony
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [September 21, 2015, 7:49pm UTC](https://discuss.elastic.co/t/how-do-we-convert-from-a-hash-to-json-before-event-reaches-the-grok-filter/29445/7 "2015-09-21T19:49:52Z")
That's actually not easily done. You might be able to use the [aggregate filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html) but I think one problem is that it's hard to know which events should be merged. I don't think one can count on the various metrics to arrive in the same order every time, they might be interspersed by other events, and events might be processed in batch so that multiple sets of related events are processed at the same time. Getting this right is probably hard with the stock plugins.
---
### Author:  [@tmo\_bgc](https://discuss.elastic.co/u/tmo_bgc)
#### Post date: [September 21, 2015, 7:55pm UTC](https://discuss.elastic.co/t/how-do-we-convert-from-a-hash-to-json-before-event-reaches-the-grok-filter/29445/8 "2015-09-21T19:55:25Z")
I see. I figured that as much but wanted to hear from an expert. If I come up with a custom solution i will post it.
---
### Author:  [@srinivaas](https://discuss.elastic.co/u/srinivaas)
#### Post date: [February 29, 2016, 9:10am UTC](https://discuss.elastic.co/t/how-do-we-convert-from-a-hash-to-json-before-event-reaches-the-grok-filter/29445/9 "2016-02-29T09:10:57Z")
Hi, Were you able to find a solution to aggregate related JMX events into single event?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 5:09am UTC](https://discuss.elastic.co/t/how-do-we-convert-from-a-hash-to-json-before-event-reaches-the-grok-filter/29445/10 "2017-07-06T05:09:23Z")