# How about standard deviation through a scripted field painless expression?

**URL:** <https://discuss.elastic.co/t/how-about-standard-deviation-through-a-scripted-field-painless-expression/189152>\
**Category:** Kibana\
**Created:** [July 5, 2019, 6:40pm UTC](https://discuss.elastic.co/t/how-about-standard-deviation-through-a-scripted-field-painless-expression/189152 "2019-07-05T18:40:17Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Joseph\_Mak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joseph_mak/32/42230_2.png) [@Joseph\_Mak](https://discuss.elastic.co/u/Joseph_Mak)\
**Post date:** [July 5, 2019, 6:40pm UTC](https://discuss.elastic.co/t/how-about-standard-deviation-through-a-scripted-field-painless-expression/189152/1 "2019-07-05T18:40:17Z")

</div>

Continuing the discussion from [Standard Deviation in Timelion](https://discuss.elastic.co/t/standard-deviation-in-timelion/131885/4):

Is Timelion able to plot a graph involving standard deviation, through a scripted field (painless expression)?

---

<div class="post-metadata">

**Author:** ![lukeelmers](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukeelmers/32/35230_2.png) [@lukeelmers](https://discuss.elastic.co/u/lukeelmers)\
**Post date:** [July 5, 2019, 6:56pm UTC](https://discuss.elastic.co/t/how-about-standard-deviation-through-a-scripted-field-painless-expression/189152/2 "2019-07-05T18:56:26Z")

</div>

Hi @Joseph_Mak,

Unfortunately I do not believe this is possible; Scripted fields in Kibana are computed on the fly for a single document, so they have access to all fields of that document, but cannot perform calculations based on multiple documents -- which is what you'd need to do to determine standard deviation.

If you're curious, [this blog post](https://www.elastic.co/blog/using-painless-kibana-scripted-fields) dives a bit deeper on the capabilities of scripted fields.

---

<div class="post-metadata">

**Author:** ![Joseph\_Mak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joseph_mak/32/42230_2.png) [@Joseph\_Mak](https://discuss.elastic.co/u/Joseph_Mak)\
**Post date:** [July 5, 2019, 8:24pm UTC](https://discuss.elastic.co/t/how-about-standard-deviation-through-a-scripted-field-painless-expression/189152/3 "2019-07-05T20:24:37Z")

</div>

Thanks @lukeelmers  
I am looking for standard deviation on a single document.

According to this link, scripted field can do mean, max, min, average, median, and sum.  
[https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-expression.html](https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-expression.html)  
A line graph can do standard deviation too. Is there way to do standard deviation?

---

<div class="post-metadata">

**Author:** ![lukeelmers](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukeelmers/32/35230_2.png) [@lukeelmers](https://discuss.elastic.co/u/lukeelmers)\
**Post date:** [July 5, 2019, 10:15pm UTC](https://discuss.elastic.co/t/how-about-standard-deviation-through-a-scripted-field-painless-expression/189152/4 "2019-07-05T22:15:45Z")

</div>

Ah, thanks for clarifying. If you have a list of numbers in a single document, you could try calculating the standard deviation manually via scripted field, but I'm not aware of any built-in ways to do this with painless.

```auto
def foo = doc['foo'];
def mean = foo.sum() / foo.length;
for (int i = 0; i < foo.length; i++) {
  // do the math
}
return result;

```

---

<div class="post-metadata">

**Author:** ![Joseph\_Mak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joseph_mak/32/42230_2.png) [@Joseph\_Mak](https://discuss.elastic.co/u/Joseph_Mak)\
**Post date:** [July 8, 2019, 6:52pm UTC](https://discuss.elastic.co/t/how-about-standard-deviation-through-a-scripted-field-painless-expression/189152/5 "2019-07-08T18:52:03Z")

</div>

Thank you! @lukeelmers but foo.length (length of list) seems to be always 1.  
So I cannot loop through a set of values on a particular time range (like for a particular day)

Example, I put the following in my index.

POST /testnumbers/doc/\_bulk  
{ "index": {"\_id":779}}  
{ "name":"One","value": 1, "@timestamp": "2019-07-07"}  
{ "index": {"\_id":780}}  
{ "name":"Two","value": 2, "@timestamp": "2019-07-07"}  
{ "index": {"\_id":781}}  
{ "name":"Three","value": 3, "@timestamp": "2019-07-07"}  
{ "index": {"\_id":782}}  
{ "name":"Four","value": 4, "@timestamp": "2019-07-07"}  
{ "index": {"\_id":784}}  
{ "name":"Five","value": 5, "@timestamp": "2019-07-07"}

and if myscriptedfield is simply return foo.length. (I'll worry about stddev and other math later)

This is my timelion:  
.es(index=testnumbers\*,timefield="@timestamp",metric=avg:myscriptfield).points().

On July 7 of Timelion graph, I expect see a cirlce of 5 on this scripted field? but I only see 1 (seems like the scripted field is only dealing with a single value).

---

<div class="post-metadata">

**Author:** ![lukeelmers](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukeelmers/32/35230_2.png) [@lukeelmers](https://discuss.elastic.co/u/lukeelmers)\
**Post date:** [July 9, 2019, 7:50pm UTC](https://discuss.elastic.co/t/how-about-standard-deviation-through-a-scripted-field-painless-expression/189152/6 "2019-07-09T19:50:45Z")

</div>

> So I cannot loop through a set of values on a particular time range (like for a particular day)

That's correct; scripted fields will only work on a per-document basis.

> On July 7 of Timelion graph, I expect see a cirlce of 5 on this scripted field? but I only see 1 (seems like the scripted field is only dealing with a single value).

Yep, this is the behavior I would expect. Each doc only has one `value`, so if your scripted field is just returning the `value.length`, it will always be `1`.

Scripted fields won't let you do scripts based on aggregations such as a date range; each script only has knowledge of the fields in the single document it is run against.

So you could only calculate the standard deviation of fields that exist within that document. In the `foo` example, it would only work if you have a list of numbers inside of your doc, e.g.

```auto
{
  foo: [12, 13, 14, 15]
}

```

or multiple fields with the numbers you want to look at:

```auto
{
  a: 12,
  b: 13,
  c: 14,
  d: 15
}

```

---

<div class="post-metadata">

**Author:** ![Joseph\_Mak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joseph_mak/32/42230_2.png) [@Joseph\_Mak](https://discuss.elastic.co/u/Joseph_Mak)\
**Post date:** [July 9, 2019, 9:50pm UTC](https://discuss.elastic.co/t/how-about-standard-deviation-through-a-scripted-field-painless-expression/189152/7 "2019-07-09T21:50:49Z")

</div>

Thank you so much @lukeelmers

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 6, 2019, 9:50pm UTC](https://discuss.elastic.co/t/how-about-standard-deviation-through-a-scripted-field-painless-expression/189152/8 "2019-08-06T21:50:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.