How about standard deviation through a scripted field painless expression?

[Joseph_Mak]

Continuing the discussion from Standard Deviation in Timelion:

Is Timelion able to plot a graph involving standard deviation, through a scripted field (painless expression)?

[lukeelmers]

Hi @Joseph_Mak,

Unfortunately I do not believe this is possible; Scripted fields in Kibana are computed on the fly for a single document, so they have access to all fields of that document, but cannot perform calculations based on multiple documents -- which is what you'd need to do to determine standard deviation.

If you're curious, this blog post dives a bit deeper on the capabilities of scripted fields.

[Joseph_Mak]

Thanks @lukeelmers
I am looking for standard deviation on a single document.

According to this link, scripted field can do mean, max, min, average, median, and sum.
https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-expression.html
A line graph can do standard deviation too. Is there way to do standard deviation?

[lukeelmers]

Ah, thanks for clarifying. If you have a list of numbers in a single document, you could try calculating the standard deviation manually via scripted field, but I'm not aware of any built-in ways to do this with painless.

def foo = doc['foo'];
def mean = foo.sum() / foo.length;
for (int i = 0; i < foo.length; i++) {
  // do the math
}
return result;

[Joseph_Mak]

Thank you! @lukeelmers but foo.length (length of list) seems to be always 1.
So I cannot loop through a set of values on a particular time range (like for a particular day)

Example, I put the following in my index.

POST /testnumbers/doc/_bulk
{ "index": {"_id":779}}
{ "name":"One","value": 1, "@timestamp": "2019-07-07"}
{ "index": {"_id":780}}
{ "name":"Two","value": 2, "@timestamp": "2019-07-07"}
{ "index": {"_id":781}}
{ "name":"Three","value": 3, "@timestamp": "2019-07-07"}
{ "index": {"_id":782}}
{ "name":"Four","value": 4, "@timestamp": "2019-07-07"}
{ "index": {"_id":784}}
{ "name":"Five","value": 5, "@timestamp": "2019-07-07"}

and if myscriptedfield is simply return foo.length. (I'll worry about stddev and other math later)

This is my timelion:
.es(index=testnumbers*,timefield="@timestamp",metric=avg:myscriptfield).points().

On July 7 of Timelion graph, I expect see a cirlce of 5 on this scripted field? but I only see 1 (seems like the scripted field is only dealing with a single value).

[lukeelmers]

So I cannot loop through a set of values on a particular time range (like for a particular day)

That's correct; scripted fields will only work on a per-document basis.

On July 7 of Timelion graph, I expect see a cirlce of 5 on this scripted field? but I only see 1 (seems like the scripted field is only dealing with a single value).

Yep, this is the behavior I would expect. Each doc only has one value, so if your scripted field is just returning the value.length, it will always be 1.

Scripted fields won't let you do scripts based on aggregations such as a date range; each script only has knowledge of the fields in the single document it is run against.

So you could only calculate the standard deviation of fields that exist within that document. In the foo example, it would only work if you have a list of numbers inside of your doc, e.g.

{
  foo: [12, 13, 14, 15]
}

or multiple fields with the numbers you want to look at:

{
  a: 12,
  b: 13,
  c: 14,
  d: 15
}

[Joseph_Mak]

Thank you so much @lukeelmers