# Display concurrency in data on Kibana

**URL:** <https://discuss.elastic.co/t/display-concurrency-in-data-on-kibana/26006>\
**Category:** Kibana\
**Created:** [July 21, 2015, 4:50pm UTC](https://discuss.elastic.co/t/display-concurrency-in-data-on-kibana/26006 "2015-07-21T16:50:44Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![EricK](https://avatars.discourse-cdn.com/v4/letter/e/bbce88/32.png) [@EricK](https://discuss.elastic.co/u/EricK)\
**Post date:** [July 21, 2015, 4:50pm UTC](https://discuss.elastic.co/t/display-concurrency-in-data-on-kibana/26006/1 "2015-07-21T16:50:44Z")

</div>

I have fields with start date and duration (seconds) representing start and duration of a phone call. I would like to define concurrency (a number) when the same destination (also a field) is active within the same duration.

That is if a ncall starts now for 30 seconds and another call to the same destination starts 15 seconds into the first call, concurrency should be 2 for the remaining 15 seconds.

How do I define this in Kibana visualization as a line graph over time?

---

<div class="post-metadata">

**Author:** ![asafyigal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asafyigal/32/685_2.png) [@asafyigal](https://discuss.elastic.co/u/asafyigal)\
**Post date:** [July 22, 2015, 7:14am UTC](https://discuss.elastic.co/t/display-concurrency-in-data-on-kibana/26006/2 "2015-07-22T07:14:00Z")

</div>

This is very difficult and probably not the right tool for the task.

In order to do this you would have to write a script withing ES that computes the concurrency by aggregating on the destination and calculate it for every event.

The easy way to do this is to write the concurrency to the log files when you record the values - so whenever a call starts check if a call already exist (and how many of them exist) and write it in the logs. Then you can easily graph it.

Hope this helps.

-- Asaf.

---

<div class="post-metadata">

**Author:** ![colings86](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colings86/32/44960_2.png) [@colings86](https://discuss.elastic.co/u/colings86)\
**Post date:** [July 24, 2015, 3:44pm UTC](https://discuss.elastic.co/t/display-concurrency-in-data-on-kibana/26006/3 "2015-07-24T15:44:52Z")

</div>

You can actually do this if you query Elasticsearch directly by using a script on a `date_histogram` aggregation which adds the call document to all the buckets from the `start_time` to `start_time + duration`. The following gist contains a Sense recreation that details what I mean: [https://gist.github.com/jpountz/cebb8452648c36099cee](https://gist.github.com/jpountz/cebb8452648c36099cee)

As for doing this in Kibana, I know it's possible to run scripts in Kibana for some actions but I'm not sure if this would be possible, maybe someone more familiar with Kibana could comment? At the very least you would probably need to have a file script for this as Kibana won't let you run inline groovy scripts.

Hope that helps

---

<div class="post-metadata">

**Author:** ![spalger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spalger/32/14092_2.png) [@spalger](https://discuss.elastic.co/u/spalger)\
**Post date:** [July 24, 2015, 3:56pm UTC](https://discuss.elastic.co/t/display-concurrency-in-data-on-kibana/26006/4 "2015-07-24T15:56:45Z")

</div>

Wow, nice script @colings86 and @jpountz

@EricK if you have scripting enabled in elasticsearch you can use it within your date\_histogram by overriding the aggregations parameters with the "JSON input" advanced config option. This would look something like the screenshot below (note the `field: null` bit which removes the field parameter from the params):

 ![](https://us1.discourse-cdn.com/elastic/original/2X/6/62c0f670d4d22d0ab89f6f910a56ffc76c7cfbf1.png)

---

<div class="post-metadata">

**Author:** ![EricK](https://avatars.discourse-cdn.com/v4/letter/e/bbce88/32.png) [@EricK](https://discuss.elastic.co/u/EricK)\
**Post date:** [July 24, 2015, 6:00pm UTC](https://discuss.elastic.co/t/display-concurrency-in-data-on-kibana/26006/5 "2015-07-24T18:00:36Z")

</div>

This is awesome!

Why are we multiplying duration by 1000? Also is "interval" a kibana variable? Lastly, can't seem to locate documentation on use of "Json input" field this way - explains the varied responses.

Any documentation pointers would be useful.

---

<div class="post-metadata">

**Author:** ![EricK](https://avatars.discourse-cdn.com/v4/letter/e/bbce88/32.png) [@EricK](https://discuss.elastic.co/u/EricK)\
**Post date:** [July 25, 2015, 12:09pm UTC](https://discuss.elastic.co/t/display-concurrency-in-data-on-kibana/26006/6 "2015-07-25T12:09:22Z")

</div>

Tried the script in Kibana and getting no results. Anything i'm doing wrong?

![](https://us1.discourse-cdn.com/elastic/original/2X/5/5bb36b639b1b52a636b40a5751b7c388b317383d.png)

---

<div class="post-metadata">

**Author:** ![tbragin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tbragin/32/45166_2.png) [@tbragin](https://discuss.elastic.co/u/tbragin)\
**Post date:** [July 25, 2015, 2:42pm UTC](https://discuss.elastic.co/t/display-concurrency-in-data-on-kibana/26006/7 "2015-07-25T14:42:24Z")

</div>

Default scripting language in Kibana is Lucene Expressions, so you have to specify {"script" : "blah", "lang" : "groovy"} in your JSON input.

You can also use static scripts, if you don't want to use dynamic - more on that here: [Calling groovy script from Kibana](https://discuss.elastic.co/t/calling-groovy-script-from-kibana/2542/3)

---

<div class="post-metadata">

**Author:** ![joris\_renkens](https://avatars.discourse-cdn.com/v4/letter/j/838e76/32.png) [@joris\_renkens](https://discuss.elastic.co/u/joris_renkens)\
**Post date:** [August 21, 2015, 10:01am UTC](https://discuss.elastic.co/t/display-concurrency-in-data-on-kibana/26006/8 "2015-08-21T10:01:36Z")

</div>

Hi,

Thank you Tanya for the link. I did all this and kibana sends the following request to elasticsearch:

```
 "aggs": {
    "3": {
      "date_histogram": {
        "field": "recordStart",
        "interval": "minute",
        "pre_zone": "+02:00",
        "pre_zone_adjust_large_interval": true,
        "min_doc_count": 1,
        "extended_bounds": {
          "min": 1414574705856,
          "max": 1415722155456
        },
        "script": "start = doc['recordStart'].value; duration = doc['recordDuration'].value; l = []; for (long i = 0; i < duration; i += 60000) { l.add(start + i); }; return l;",
        "lang": "groovy"
      }
    }
  }

```

This query returns an error. When I remove the "field" property and rerun it in elasticsearch it works fine. Is this a bug in kibana that it adds the "field" property as well as the script? Is there a workaround?

---

<div class="post-metadata">

**Author:** ![tbragin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tbragin/32/45166_2.png) [@tbragin](https://discuss.elastic.co/u/tbragin)\
**Post date:** [August 21, 2015, 3:34pm UTC](https://discuss.elastic.co/t/display-concurrency-in-data-on-kibana/26006/9 "2015-08-21T15:34:50Z")

</div>

What kind of error do you get?

---

<div class="post-metadata">

**Author:** ![joris\_renkens](https://avatars.discourse-cdn.com/v4/letter/j/838e76/32.png) [@joris\_renkens](https://discuss.elastic.co/u/joris_renkens)\
**Post date:** [August 21, 2015, 5:44pm UTC](https://discuss.elastic.co/t/display-concurrency-in-data-on-kibana/26006/10 "2015-08-21T17:44:42Z")

</div>

```
{

```

"error": "SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[NZCzFCLeQiOQrSjvhSWi8A][am][0]: QueryPhaseExecutionException[[am][0]: query[ConstantScore(_:_)],from[0],size[10]: Query Failed [Failed to execute main query]]; nested: ClassCastException[java.util.ArrayList cannot be cast to java.lang.Number]; }{[NZCzFCLeQiOQrSjvhSWi8A][am][1]: QueryPhaseExecutionException[[am][1]: query[ConstantScore(_:_)],from[0],size[10]: Query Failed [Failed to execute main query]]; nested: ClassCastException[java.util.ArrayList cannot be cast to java.lang.Number]; }{[NZCzFCLeQiOQrSjvhSWi8A][am][2]: QueryPhaseExecutionException[[am][2]: query[ConstantScore(_:_)],from[0],size[10]: Query Failed [Failed to execute main query]]; nested: ClassCastException[java.util.ArrayList cannot be cast to java.lang.Number]; }{[NZCzFCLeQiOQrSjvhSWi8A][am][3]: QueryPhaseExecutionException[[am][3]: query[ConstantScore(_:_)],from[0],size[10]: Query Failed [Failed to execute main query]]; nested: ClassCastException[java.util.ArrayList cannot be cast to java.lang.Number]; }{[NZCzFCLeQiOQrSjvhSWi8A][am][4]: QueryPhaseExecutionException[[am][4]: query[ConstantScore(_:_)],from[0],size[10]: Query Failed [Failed to execute main query]]; nested: ClassCastException[java.util.ArrayList cannot be cast to java.lang.Number]; }]",  
"status": 500  
}

---

<div class="post-metadata">

**Author:** ![tbragin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tbragin/32/45166_2.png) [@tbragin](https://discuss.elastic.co/u/tbragin)\
**Post date:** [August 22, 2015, 1:28pm UTC](https://discuss.elastic.co/t/display-concurrency-in-data-on-kibana/26006/11 "2015-08-22T13:28:50Z")

</div>

> [@spalger](#):
>
> (note the field: null bit which removes the field parameter from the params):

In re-reading Spencer's post below, he suggests an additional parameter to the script to remove the field property. I see that you're missing that in your script. Could you try adding that and see what happens?

---

<div class="post-metadata">

**Author:** ![joris\_renkens](https://avatars.discourse-cdn.com/v4/letter/j/838e76/32.png) [@joris\_renkens](https://discuss.elastic.co/u/joris_renkens)\
**Post date:** [August 23, 2015, 6:45am UTC](https://discuss.elastic.co/t/display-concurrency-in-data-on-kibana/26006/12 "2015-08-23T06:45:28Z")

</div>

I also tried this but it doesn't remove the field parameter as promised. With the following json input:

```
{
"interval":"minute",
"script": "start = doc['recordStart'].value; duration = doc['recordDuration'].value; l = []; for (long i = 0; i < duration; i += 60000) { l.add(start + i); }; return l;",
"field":null,
"lang": "groovy"

```

}

I get the following request

```
 "aggs": {
"2": {
  "date_histogram": {
    "field": null,
    "interval": "minute",
    "pre_zone": "+02:00",
    "pre_zone_adjust_large_interval": true,
    "min_doc_count": 1,
    "extended_bounds": {
      "min": 1366292067758,
      "max": 1433416997446
    },
    "script": "start = doc['recordStart'].value; duration = doc['recordDuration'].value; l = []; for (long i = 0; i < duration; i += 60000) { l.add(start + i); }; return l;",
    "lang": "groovy"
  }
}

```

}

Which gives the following error

```
Error: Request to Elasticsearch failed: (big part removed because message was too long)

```

Failure [Unexpected token VALUE\_NULL in [2].]]; }]"}  
at [http://localhost:5601/index.js?\_b=7489:43092:38](http://localhost:5601/index.js?_b=7489:43092:38)  
at Function.Promise.try ([http://localhost:5601/index.js?\_b=7489:46434:26](http://localhost:5601/index.js?_b=7489:46434:26))  
at [http://localhost:5601/index.js?\_b=7489:46412:27](http://localhost:5601/index.js?_b=7489:46412:27)  
at Array.map (native)  
at Function.Promise.map ([http://localhost:5601/index.js?\_b=7489:46411:30](http://localhost:5601/index.js?_b=7489:46411:30))  
at callResponseHandlers ([http://localhost:5601/index.js?\_b=7489:43064:22](http://localhost:5601/index.js?_b=7489:43064:22))  
at [http://localhost:5601/index.js?\_b=7489:43182:16](http://localhost:5601/index.js?_b=7489:43182:16)  
at wrappedCallback ([http://localhost:5601/index.js?\_b=7489:20893:81](http://localhost:5601/index.js?_b=7489:20893:81))  
at wrappedCallback ([http://localhost:5601/index.js?\_b=7489:20893:81](http://localhost:5601/index.js?_b=7489:20893:81))  
at [http://localhost:5601/index.js?\_b=7489:20979:26](http://localhost:5601/index.js?_b=7489:20979:26)

---

<div class="post-metadata">

**Author:** ![tbragin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tbragin/32/45166_2.png) [@tbragin](https://discuss.elastic.co/u/tbragin)\
**Post date:** [August 28, 2015, 4:05am UTC](https://discuss.elastic.co/t/display-concurrency-in-data-on-kibana/26006/13 "2015-08-28T04:05:00Z")

</div>

It worked for me with the following data and config:

```
curl -XPOST 'http://localhost:9200/test3' -d '{
    "settings" : {
        "number_of_shards" : 1
    },
    "mappings" : {
        "test" : {
            "properties" : {
                "start_date" : { "type" : "date"},
                "end_date" : { "type" : "date"},
                "duration" : { "type" : "integer"}
            }
        }
    }
}'

curl -XPUT 'http://localhost:9200/test3/test/1' -d '{
    "start_date" : "2015-08-23T00:01:00",
    "end_date" : "2015-08-23T14:02:00",
    "duration" : 1
}'

curl -XPUT 'http://localhost:9200/test3/test/2' -d '{
    "start_date" : "2015-08-23T00:01:30",
    "end_date" : "2015-08-23T14:02:30",
    "duration" : 1
}'

curl -XPUT 'http://localhost:9200/test3/test/3' -d '{
    "start_date" : "2015-08-23T00:01:45",
    "end_date" : "2015-08-23T14:02:45",
    "duration" : 1
}'

curl -XPUT 'http://localhost:9200/test3/test/4' -d '{
    "start_date" : "2015-08-23T00:01:55",
    "end_date" : "2015-08-23T14:02:55",
    "duration" : 1
}'

curl -XPUT 'http://localhost:9200/test3/test/5' -d '{
    "start_date" : "2015-08-23T00:02:00",
    "end_date" : "2015-08-23T14:04:00",
    "duration" : 1
}'

```

Script:

```
{
"interval":"minute",
"script": "start = doc['start_date'].value; duration = doc['duration'].value*1000*1000; l = []; for (long i = 0; i < duration; i += 60000) { l.add(start + i); }; return l;",
"field": null,
"lang": "groovy"
}

```

 ![](https://us1.discourse-cdn.com/elastic/original/2X/6/69938f0a7491cc422646409d2712ddbf31c5fded.png)

---

<div class="post-metadata">

**Author:** ![joris\_renkens](https://avatars.discourse-cdn.com/v4/letter/j/838e76/32.png) [@joris\_renkens](https://discuss.elastic.co/u/joris_renkens)\
**Post date:** [August 28, 2015, 6:55am UTC](https://discuss.elastic.co/t/display-concurrency-in-data-on-kibana/26006/14 "2015-08-28T06:55:23Z")

</div>

Very strange, I've followed your case with exactly the same data and config and my kibana throws the same error I mentioned earlier. I'm using:

Kibana server: 4.1.1  
Elasticsearch: 1.7.1

Is it possible that I've got some settings wrong?

---

<div class="post-metadata">

**Author:** ![tbragin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tbragin/32/45166_2.png) [@tbragin](https://discuss.elastic.co/u/tbragin)\
**Post date:** [August 28, 2015, 1:36pm UTC](https://discuss.elastic.co/t/display-concurrency-in-data-on-kibana/26006/15 "2015-08-28T13:36:49Z")

</div>

Ah - I'm running ES 2.0 beta 1 (just released yesterday) and a snapshot build of Kibana 4.2 (which you can get at the bottom of the page [here](https://github.com/elastic/kibana), and should be released as 4.2 beta soon).

When I run this on ES 1.6 and Kibana 4.1.1 I get the same error as you. Let me see if I can get some clarification on that.

 ![](https://us1.discourse-cdn.com/elastic/original/2X/c/c330cda34eddfc741676060e786365f9a9232979.png)

---

<div class="post-metadata">

**Author:** ![tbragin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tbragin/32/45166_2.png) [@tbragin](https://discuss.elastic.co/u/tbragin)\
**Post date:** [August 28, 2015, 5:44pm UTC](https://discuss.elastic.co/t/display-concurrency-in-data-on-kibana/26006/16 "2015-08-28T17:44:37Z")

</div>

I confirmed that the "field": null feature was introduced in the Kibana 4.2 timeframe, so that is why this is not working in Kibana 4.1.1. @joris_renkens assuming you're running in test / dev, would it be possible for you to use ES 2.0 beta and Kibana 4.2 beta for this project?

---

<div class="post-metadata">

**Author:** ![joris\_renkens](https://avatars.discourse-cdn.com/v4/letter/j/838e76/32.png) [@joris\_renkens](https://discuss.elastic.co/u/joris_renkens)\
**Post date:** [August 31, 2015, 2:52pm UTC](https://discuss.elastic.co/t/display-concurrency-in-data-on-kibana/26006/17 "2015-08-31T14:52:53Z")

</div>

It worked! Do you have any idea when ES 2.0 and Kibana 4.2 will become stable?

---

<div class="post-metadata">

**Author:** ![tbragin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tbragin/32/45166_2.png) [@tbragin](https://discuss.elastic.co/u/tbragin)\
**Post date:** [August 31, 2015, 5:44pm UTC](https://discuss.elastic.co/t/display-concurrency-in-data-on-kibana/26006/18 "2015-08-31T17:44:50Z")

</div>

Official Kibana 4.2.0-beta1 was released today: [https://www.elastic.co/blog/kibana-4-2-beta-1-i-heard-you-like-betas](https://www.elastic.co/blog/kibana-4-2-beta-1-i-heard-you-like-betas)

GA for ES 2.0 and Kibana 4.2 depends a bit on the kind of feedback we get from the community and whether any significant changes are required, but we are currently targeting it to early fall 🙂

---

<div class="post-metadata">

**Author:** ![Cristian](https://avatars.discourse-cdn.com/v4/letter/c/a698b9/32.png) [@Cristian](https://discuss.elastic.co/u/Cristian)\
**Post date:** [February 5, 2017, 9:52pm UTC](https://discuss.elastic.co/t/display-concurrency-in-data-on-kibana/26006/19 "2017-02-05T21:52:10Z")

</div>

Hi,  
Im sorry to bump this old thread. I think I have a similar problem.

I have fields "Start Timestamp" , "End Timestamp", "Status", and I would like to create a daily date histogram and count how many transactions there are per day per status. The duration of a transaction could be anything from one hour to 2-3 weeks so if a transaction of lets say 2 days should be included in the calculation in 2 bars. I dont know how to proceed with this. Any suggestions? (im using ES 5.2)  
Br  
Cristian

---

<div class="post-metadata">

**Author:** ![benpolzin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benpolzin/32/12408_2.png) [@benpolzin](https://discuss.elastic.co/u/benpolzin)\
**Post date:** [February 10, 2017, 4:09pm UTC](https://discuss.elastic.co/t/display-concurrency-in-data-on-kibana/26006/20 "2017-02-10T16:09:48Z")

</div>

I'm in a similar situation except have start & end time instead of start & duration. I got the previous examples working in current versions of ES (5.2) with updated syntax, but.. Groovy is deprecated and requires enabling in elasticsearch.yml and then still throws log warnings about being deprecated.

[details=Groovy script for 5.2]{  
GET test/\_search  
{  
"aggs": {  
"my\_histo": {  
"date\_histogram": {  
"script": {  
"lang": "groovy",  
"inline": "start = doc['start\_date'].value; duration = doc['duration'].value_1000_1000; l = []; for (long i = 0; i \< duration; i += 60000) { l.add(start + i); }; return l;"  
},  
"interval": "minute"  
}  
}  
}  
}  
}[/details]

So... anyone know if this is now possible in Timelion? It feels like it should be, but I'm just diving into Timelion for the first time and not finding a function that maps to this off the bat.

Or if not possible in Timelion yet then time to move on to Painless?

[Next page](https://discuss.elastic.co/t/display-concurrency-in-data-on-kibana/26006.md?page=2)