Logstash filtering nested JSON fields (mutate / grok)

Elastic Stack Logstash
1

Hello Folks,

input {
  http_poller {
    urls => {
      test2 => {
        method => get
        url => "http://www.coincap.io/history/1day/XRP"
      }
    }
    request_timeout => 60
    schedule => { cron => "* * * * * * UTC"}
    codec => "json"
  }
}
output {
  stdout {
    codec => rubydebug
  }
}

I am fetching a JSON data from the above API. The logstash rubydebug output of that JSON is as below:

{
             "preminedSig" => false,
                   "isBuy" => true,
                     "dom" => 39,
                  "altCap" => 69121887494,
             "explorerURL" => "https://blockexplorer.com/",
                  "btcCap" => 44395178760,
             "vwapDataBTC" => "2682.6451919194055",
              "capPercent" => "5.70",
                  "supply" => "16389000",
                    "long" => "Bitcoin",
    "cap24hrChangePercent" => "5.70",
                 "twitter" => "",
                    "perc" => "5.70",
              "market_cap" => [
        [   0] [
            [0] 1367174841000,
            [1] 1500517590
        ],
        [   1] [
            [0] 1367261101000,
            [1] 1575032004
        ],
		],
		    "@timestamp" => 2017-06-13T20:02:21.397Z,
		         "price" => [
		        [  0] [
		            [0] 1497286765000,
		            [1] 0.255964
		        ],
		        [  1] [
		            [0] 1497287042000,
		            [1] 0.255944
		        ],
		        [  2] [
		            [0] 1497287359000,
		            [1] 0.253767
		        ]
}

In the above JSON output , i need to parse out the price field which contains two values ([0] , [1] ). One is Timestamp and the other is price. i need to fetch it out those two fields and set the type for one field as TimeStamp and other as number and send to Elastic Search.

I have tried using Mutate and Grok filter. I am unable to attain the result.
Can anyone of guys help me out ?

[ 1] [
[0] 1497287042000, ------- Give a seperate field name for this as timestamp
[1] 0.255944 -- give a seperate field name for this as price.
],

2

Okay, it's clear what you want to do with a single (timestamp, tuple) entity, but the price field contains three such tuples. Do you want to process just one of them or all of them? How?

1 Like
3

I need all the data in the price . from price [0] --- price[n] ... each price has two internal fields right. that i want to segregate one as timestamp field and other as a sub_price fields like that. giving a name to it.

Like this.

price [0] = { "timestamp" : 1497286765000(data type to timestamp) , "sub_price":0.2555}.
4

Use a ruby filter. Something like this should work:

event.set('price', event.get('price').collect { |p| {"timestamp" => p[0], "sub_price" => p[1]} })

To turn the timestamp field into a date field in ES you have a couple of options depending on what you want the documents to look like; you can store either an epoch number and configure the index's mappings to treat it as a date or you can convert the epoch into a human-readable timestamp string and have ES's automapper automatically guess that it's a date.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.