Conccurency in data

[makos63]

Hello,
we want to see what was the maximum and the minimum of parallel event we had in given interval. I know that is possible in Splunk.

I tried this already, but could not rewrite Groovy script in Painless successfully

Display concurrency in data on Kibana

I have fields with start date and duration (seconds) representing start and duration of a phone call. I would like to define concurrency (a number) when the same destination (also a field) is active within the same duration. That is if a ncall starts now for 30 seconds and another call to the same destination starts 15 seconds into the first call, concurrency should be 2 for the remaining 15 seconds. How do I define this in Kibana visualization as a line graph over time?

This is how we did it in Splunk:

sourcetype="log" ("EXPECTED:Connected" OR "EXPECTED: Disconnected") | rex field=_raw "SessionID":"(?<dial_session>[^"]+)" | transaction host,dial_session startswith=AVPEventConnected endswith=AVPEventDisconnected maxspan=10m | concurrency duration=duration | timechart min(concurrency) AS "Min Concurrency", max(concurrency) AS "Max Concurrency" span=200s

We really would love to replace Splunk with Elasticsearch and Kibana but we need to be sure that most of Splunk can is doable with elasticsearch.

[Mark_Harwood]

Hi makos63,

The range field is designed for this sort of work but currently does not support aggregations of the sort used in Kibana.
See related discussion here

[makos63]

So there is no method to automate this process, i need to manully create data ranges in Kibana. Still i do not understand how can i retrieve all events which were active in specified data ranges.

[Mark_Harwood]

@makos63

Still i do not understand how can i retrieve all events which were active in specified data ranges.

You could certainly use a single range search to match docs with a single range field.
However for the purposes of aggregations and filling in bars on a Kibana histogram you'd need those docs to have a different regular date field but with an array of values. The gap between each value would have to be whatever makes sense for your visualisation e.g. weeks or days.

[makos63]

The idea worked! Thank you! It may need some more tweeking, however for now looks good enough.

Here, this is how i made it work:

PUT test2
{
"mappings": {
"transaciton": {
"properties": {
"timestamp" : { "type" : "date"},
"sessionID": { "type": "integer" },
"connectAt" : {"type" : "date"},
"disconnectAt" : {"type" : "date"},
"duration": { "type": "date" },
"message": { "type": "text",
"fielddata": true},
"active":{"type":"boolean"}

}
}

PUT test2/transaciton/1
{
"timestamp":"2019-06-12T12:00:01Z",
"sessionID": "1",
"connectAt":"2019-06-12T12:00:00Z",
"disconnectAt":"2019-06-12T12:01:30Z",
"durations":[
{"duration":"2019-06-12T12:00:00Z"},
{"duration":"2019-06-12T12:00:30Z"},
{"duration":"2019-06-12T12:01:00Z"},
{"duration":"2019-06-12T12:01:30Z"}
],
"message":"SUCCEED",
"active":false
}

PUT test2/transaciton/2
{
"timestamp":"2019-06-12T12:01:01Z",
"sessionID": "2",
"connectAt":"2019-06-12T12:01:00Z",
"disconnectAt":"2019-06-12T12:01:30Z",
"durations":[
{"stamp":"2019-06-12T12:01:00Z"},
{"stamp":"2019-06-12T12:01:30Z"}
],
"message":"SUCCEED",
"active":false
}
PUT test2/transaciton/3
{
"timestamp":"2019-06-12T11:59:01Z",
"sessionID": "3",
"connectAt":"2019-06-12T11:59:00Z",
"disconnectAt":"2019-06-12T12:01:00Z",
"durations":[
{"stamp":"2019-06-12T11:59:00Z"},
{"stamp":"2019-06-12T11:59:30Z"},
{"stamp":"2019-06-12T12:01:00Z"}
],
"message":"SUCCEED",
"active":false
}
PUT test2/transaciton/4
{
"timestamp":"2019-06-12T12:00:31Z",
"sessionID": "4",
"connectAt":"2019-06-12T12:00:30Z",
"disconnectAt":"2019-06-12T12:02:00Z",
"durations":[
{"stamp":"2019-06-12T12:00:00Z"},
{"stamp":"2019-06-12T12:00:30Z"},
{"stamp":"2019-06-12T12:01:00Z"},
{"stamp":"2019-06-12T12:01:30Z"},
{"stamp":"2019-06-12T12:02:00Z"}
],
"message":"SUCCEED",
"active":false
}
PUT test2/transaciton/5
{
"timestamp":"2019-06-12T12:01:21Z",
"sessionID": "5",
"connectAt":"2019-06-12T12:01:20Z",
"disconnectAt":"2019-06-12T12:03:20Z",
"durations":[
{"stamp":"2019-06-12T12:01:20Z"},
{"stamp":"2019-06-12T12:01:50Z"},
{"stamp":"2019-06-12T12:02:20Z"},
{"stamp":"2019-06-12T12:02:50Z"},
{"stamp":"2019-06-12T12:03:20Z"}
],
"message":"SUCCEED",
"active":false
}
PUT test2/transaciton/6
{
"sessionID": "6",
"connectAt":"2019-06-12T12:05:20Z",
"disconnectAt":"2019-06-12T12:07:20Z",
"durations":[
{"stamp":"2019-06-12T12:05:20Z"},
{"stamp":"2019-06-12T12:05:50Z"},
{"stamp":"2019-06-12T12:06:20Z"},
{"stamp":"2019-06-12T12:06:50Z"},
{"stamp":"2019-06-12T12:07:20Z"}
],
"message":"SUCCEED",
"active":false
}

PUT test2/transaciton/7
{
"timestamp":"2019-06-12T12:15:21Z",
"sessionID": "7",
"connectAt":"2019-06-12T12:15:20Z",
"disconnectAt":"2019-06-12T12:17:20Z",
"durations":[
{"stamp":"2019-06-12T12:15:20Z"},
{"stamp":"2019-06-12T12:15:50Z"},
{"stamp":"2019-06-12T12:16:20Z"},
{"stamp":"2019-06-12T12:16:50Z"},
{"stamp":"2019-06-12T12:17:20Z"}
],
"message":"SUCCEED",
"active":false
}

[Mark_Harwood]

Glad it worked for you