Timelion - subtract by split data field (group per field before operation)

Elastic Stack Kibana
1

Just found out I have this great tool for time-series in Kibana for ES timed data and would like to analyse through >1M of the following doc type per MarketName field:

{
			"MarketName" : "BTC-XRP",
			"High" : 0.00000919,
			"Low" : 0.00000820,
			"Volume" : 74339.61396015,
			"Last" : 0.00000820,
			"BaseVolume" : 0.64966963,
			"TimeStamp" : "2014-07-09T07:19:30.15",
			"Bid" : 0.00000820,
			"Ask" : 0.00000831,
			"OpenBuyOrders" : 15,
			"OpenSellOrders" : 15,
			"PrevDay" : 0.00000821,
			"Created" : "2014-03-20T06:00:00",
			"DisplayMarketName" : null
		}

I have time-series data for 200 MarketNames for each point in time and would like to perform operations per this field, like subtraction, division, followed by 1st and 2nd degree derivatives, mvavg's, etc as per Rashid's videos :slight_smile:

My intention is to use this time series to compute per MarketName data in scenarios such as:

BaseVolume - BaseVolume(offset=-1d) or
OpenBuyOrders - OpenSellOrders, all per the same MarketName

Basically I intend to use the split in the subtracted datasource by the same field, such as:

.es(metric=max:BaseVolume, split=MarketName:10).subtract(.es(metric=max:BaseVolume, offset=-1d, split=MarketName:10)) (notice the split in the subtract) which, of course, is not working.

Now, I was wondering:

  1. what kind of aggregation should I use to split the data by MarketName after (or even before) an operation?
  2. if I use the split in the first data source, it will subtract the entire series from the per Market data
  3. should I use some other aggregations of ES before using TimeLion to achieve this kind of calculations?
  4. which would be the recommended way to perform such operations on "multidimensional" data?
  5. if I perform the subtract, is it correct to split after creating a group, if possible?

My intent is to have graphs sorted by the result of such operations (subtract, sum/divide, etc) applied for each MarketName and display/graph the first N results (as split does) but my brain is stuck on bucketing/aggregating the data.

Any suggestion will be greatly appreciated as I struggled without any success in the past couple of days.

Thanks in advance for any advice!

2

Hi @eurobosch,

applying the arithmetic operations to multiple series will be supported starting with Kibana 6.2 (see GitHub PR #14891). Sorry to hear you had to struggle with this and thanks for the detailed description of your use-case.

3

Thanks for the good news, @weltenwort, looking forward for the new version: I looked in the PR and it's exactly what I need! Worried though that I have to upgrade from 5.5 :slight_smile:

Any idea what's the 6.2 release date?

4

I can't give you a specific date :innocent:, but given our release timings it should be within the next two months. :wink:

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.