Conccurency in data

Hello,
we want to see what was the maximum and the minimum of parallel event we had in given interval. I know that is possible in Splunk.

I tried this already, but could not rewrite Groovy script in Painless successfully


This is how we did it in Splunk:

sourcetype="log" ("EXPECTED:Connected" OR "EXPECTED: Disconnected") | rex field=_raw "SessionID":"(?<dial_session>[^"]+)" | transaction host,dial_session startswith=AVPEventConnected endswith=AVPEventDisconnected maxspan=10m | concurrency duration=duration | timechart min(concurrency) AS "Min Concurrency", max(concurrency) AS "Max Concurrency" span=200s

We really would love to replace Splunk with Elasticsearch and Kibana but we need to be sure that most of Splunk can is doable with elasticsearch.

Hi makos63,

The range field is designed for this sort of work but currently does not support aggregations of the sort used in Kibana.
See related discussion here

So there is no method to automate this process, i need to manully create data ranges in Kibana. Still i do not understand how can i retrieve all events which were active in specified data ranges.

You could certainly use a single range search to match docs with a single range field.
However for the purposes of aggregations and filling in bars on a Kibana histogram you'd need those docs to have a different regular date field but with an array of values. The gap between each value would have to be whatever makes sense for your visualisation e.g. weeks or days.

The idea worked! Thank you! It may need some more tweeking, however for now looks good enough.

Here, this is how i made it work:

PUT test2
{
"mappings": {
"transaciton": {
"properties": {
"timestamp" : { "type" : "date"},
"sessionID": { "type": "integer" },
"connectAt" : {"type" : "date"},
"disconnectAt" : {"type" : "date"},
"duration": { "type": "date" },
"message": { "type": "text",
"fielddata": true},
"active":{"type":"boolean"}

}
}

PUT test2/transaciton/1
{
"timestamp":"2019-06-12T12:00:01Z",
"sessionID": "1",
"connectAt":"2019-06-12T12:00:00Z",
"disconnectAt":"2019-06-12T12:01:30Z",
"durations":[
{"duration":"2019-06-12T12:00:00Z"},
{"duration":"2019-06-12T12:00:30Z"},
{"duration":"2019-06-12T12:01:00Z"},
{"duration":"2019-06-12T12:01:30Z"}
],
"message":"SUCCEED",
"active":false
}

PUT test2/transaciton/2
{
"timestamp":"2019-06-12T12:01:01Z",
"sessionID": "2",
"connectAt":"2019-06-12T12:01:00Z",
"disconnectAt":"2019-06-12T12:01:30Z",
"durations":[
{"stamp":"2019-06-12T12:01:00Z"},
{"stamp":"2019-06-12T12:01:30Z"}
],
"message":"SUCCEED",
"active":false
}
PUT test2/transaciton/3
{
"timestamp":"2019-06-12T11:59:01Z",
"sessionID": "3",
"connectAt":"2019-06-12T11:59:00Z",
"disconnectAt":"2019-06-12T12:01:00Z",
"durations":[
{"stamp":"2019-06-12T11:59:00Z"},
{"stamp":"2019-06-12T11:59:30Z"},
{"stamp":"2019-06-12T12:01:00Z"}
],
"message":"SUCCEED",
"active":false
}
PUT test2/transaciton/4
{
"timestamp":"2019-06-12T12:00:31Z",
"sessionID": "4",
"connectAt":"2019-06-12T12:00:30Z",
"disconnectAt":"2019-06-12T12:02:00Z",
"durations":[
{"stamp":"2019-06-12T12:00:00Z"},
{"stamp":"2019-06-12T12:00:30Z"},
{"stamp":"2019-06-12T12:01:00Z"},
{"stamp":"2019-06-12T12:01:30Z"},
{"stamp":"2019-06-12T12:02:00Z"}
],
"message":"SUCCEED",
"active":false
}
PUT test2/transaciton/5
{
"timestamp":"2019-06-12T12:01:21Z",
"sessionID": "5",
"connectAt":"2019-06-12T12:01:20Z",
"disconnectAt":"2019-06-12T12:03:20Z",
"durations":[
{"stamp":"2019-06-12T12:01:20Z"},
{"stamp":"2019-06-12T12:01:50Z"},
{"stamp":"2019-06-12T12:02:20Z"},
{"stamp":"2019-06-12T12:02:50Z"},
{"stamp":"2019-06-12T12:03:20Z"}
],
"message":"SUCCEED",
"active":false
}
PUT test2/transaciton/6
{
"sessionID": "6",
"connectAt":"2019-06-12T12:05:20Z",
"disconnectAt":"2019-06-12T12:07:20Z",
"durations":[
{"stamp":"2019-06-12T12:05:20Z"},
{"stamp":"2019-06-12T12:05:50Z"},
{"stamp":"2019-06-12T12:06:20Z"},
{"stamp":"2019-06-12T12:06:50Z"},
{"stamp":"2019-06-12T12:07:20Z"}
],
"message":"SUCCEED",
"active":false
}

PUT test2/transaciton/7
{
"timestamp":"2019-06-12T12:15:21Z",
"sessionID": "7",
"connectAt":"2019-06-12T12:15:20Z",
"disconnectAt":"2019-06-12T12:17:20Z",
"durations":[
{"stamp":"2019-06-12T12:15:20Z"},
{"stamp":"2019-06-12T12:15:50Z"},
{"stamp":"2019-06-12T12:16:20Z"},
{"stamp":"2019-06-12T12:16:50Z"},
{"stamp":"2019-06-12T12:17:20Z"}
],
"message":"SUCCEED",
"active":false
}


25

1 Like

Glad it worked for you