# Trying to run the 'Get duration by using bucket script' but not working

**URL:** <https://discuss.elastic.co/t/trying-to-run-the-get-duration-by-using-bucket-script-but-not-working/269648>\
**Category:** Elasticsearch\
**Tags:** painless, transforms\
**Created:** [April 8, 2021, 11:13pm UTC](https://discuss.elastic.co/t/trying-to-run-the-get-duration-by-using-bucket-script-but-not-working/269648 "2021-04-08T23:13:14Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![gentle\_ghost](https://avatars.discourse-cdn.com/v4/letter/g/ea5d25/32.png) [@gentle\_ghost](https://discuss.elastic.co/u/gentle_ghost)\
**Post date:** [April 8, 2021, 11:13pm UTC](https://discuss.elastic.co/t/trying-to-run-the-get-duration-by-using-bucket-script-but-not-working/269648/1 "2021-04-08T23:13:14Z")

</div>

Hello,

I'm attempting to calculate Session Duration by using the provided example here:

> **[Painless examples for transforms | Elasticsearch Guide \[master\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/master/transform-painless-examples.html)**

I even followed this users successful implementation: [Computing session durations from timestamps scattered over two documents - #3 by seanowl](https://discuss.elastic.co/t/computing-session-durations-from-timestamps-scattered-over-two-documents/234127/3)

But I seem to be running into an issue. I can't get the field to populate for that actual session duration. Here's what I have in the edit json config:

```auto
  "group_by": {
    "IP": {
      "terms": {
        "field": "client_ip"
      }
    }
  },
  "aggregations": {
    "@timestamp_max": {
      "max": {
        "field": "@timestamp"
      }
    },
    "@timestamp_min": {
      "min": {
        "field": "@timestamp"
      }
    },
    "calculated_duration": {
      "bucket_script": {
        "buckets_path": {
          "min": "@timestamp_min",
          "max": "@timestamp_max"
        },
        "script": "(params.max - params.min)/1000"
      }
    }
  }
},
"description": "calculated duration",
  "dest": {
    "index": "test_duration_index"
  },
  "frequency": "1m",
  "sync": {
  "time": {
    "field": "@timestamp",
    "delay": "10s"
  }
}
}

```

The result is I get the max and min but I don't get a field created for the field calculated\_duration:

![missing_column - Copy (2)](https://us1.discourse-cdn.com/elastic/original/3X/a/0/a062f43c93705380f787ced465c7d9552e10d9a1.png)

Is there something I'm missing here?

Thanks

---

<div class="post-metadata">

**Author:** ![James\_Gowdy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_gowdy/32/74175_2.png) [@James\_Gowdy](https://discuss.elastic.co/u/James_Gowdy)\
**Post date:** [April 9, 2021, 7:51am UTC](https://discuss.elastic.co/t/trying-to-run-the-get-duration-by-using-bucket-script-but-not-working/269648/2 "2021-04-09T07:51:44Z")

</div>

Hi,  
It looks like this might be an issue with the UI rather than the actual transform preview.  
if you run the preview manually, the `_preview` api returns the correct fields.  
e.g. in kibana's Dev Tools:

```auto
POST _transform/_preview
{
  "source": {
    "index": [
      "farequote-*"
    ]
  },
  "pivot": {
    "group_by": {
      "airline": {
        "terms": {
          "field": "airline"
        }
      }
    },
    "aggregations": {
      "@timestamp_max": {
        "max": {
          "field": "@timestamp"
        }
      },
      "@timestamp_min": {
        "min": {
          "field": "@timestamp"
        }
      },
      "calculated_duration": {
        "bucket_script": {
          "buckets_path": {
            "min": "@timestamp_min",
            "max": "@timestamp_max"
          },
          "script": "(params.max - params.min)/1000"
        }
      }
    }
  }
}

```

Gives the response:

```auto
{
  "preview" : [
    {
      "@timestamp_min" : "2019-02-07T00:00:00.000Z",
      "@timestamp_max" : "2019-02-11T23:59:02.000Z",
      "calculated_duration" : 431942.0,
      "airline" : "AAL"
    },
    {
      "@timestamp_min" : "2019-02-07T00:00:00.000Z",
      "@timestamp_max" : "2019-02-11T23:59:45.000Z",
      "calculated_duration" : 431985.0,
      "airline" : "ACA"
    },
    ......

```

I've created a new issue to cover this.

> <https://github.com/elastic/kibana/issues/96688>
>
> Manually edit the Pivot configuration object JSON and add a new script agg:
> {
> "group\_by": {
> "airline": {
> "terms": {
> "field": "airline"
> ...

This UI issue should not affect the actual creation of the transform.

Cheers,  
James

---

<div class="post-metadata">

**Author:** ![gentle\_ghost](https://avatars.discourse-cdn.com/v4/letter/g/ea5d25/32.png) [@gentle\_ghost](https://discuss.elastic.co/u/gentle_ghost)\
**Post date:** [April 9, 2021, 3:57pm UTC](https://discuss.elastic.co/t/trying-to-run-the-get-duration-by-using-bucket-script-but-not-working/269648/3 "2021-04-09T15:57:55Z")

</div>

Hello,

This worked, thank you very much. I am testing this and I realize I am close to my goal but this is not exactly what I'm looking for.

I'm trying to calculate session duration from logs that do not have a listen duration. This is getting close but of course if I just do IP as a group by my "session duration" will be really high.

Do you have any way to try and calculate session duration from log events,. similar to google analytics. I know they use a timeout interval to track a session. This could be more in-depth as I'm thinking about it, would require a way to only track incoming event data to add to a previous session.

---

<div class="post-metadata">

**Author:** ![James\_Gowdy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_gowdy/32/74175_2.png) [@James\_Gowdy](https://discuss.elastic.co/u/James_Gowdy)\
**Post date:** [April 12, 2021, 7:55am UTC](https://discuss.elastic.co/t/trying-to-run-the-get-duration-by-using-bucket-script-but-not-working/269648/4 "2021-04-12T07:55:53Z")

</div>

Hi,  
I'm not aware of a capability in elasticsearch to automatically calculate session duration from a log file.  
If the data also contains a session ID or something similar, I would group by that rather than IP.

Cheers,  
James

---

<div class="post-metadata">

**Author:** ![gentle\_ghost](https://avatars.discourse-cdn.com/v4/letter/g/ea5d25/32.png) [@gentle\_ghost](https://discuss.elastic.co/u/gentle_ghost)\
**Post date:** [April 16, 2021, 4:28pm UTC](https://discuss.elastic.co/t/trying-to-run-the-get-duration-by-using-bucket-script-but-not-working/269648/5 "2021-04-16T16:28:21Z")

</div>

I've watched this video for entity centric indexing by Mark Harwood and see benefit in this. I'm trying to run the attached example code provided: [Entity-Centric Indexing - Mark Harwood | Elastic Videos](https://www.elastic.co/videos/entity-centric-indexing-mark-harwood) but running into some issues with the python script. I'm being thrown this error:

```
ElasticsearchWarning: [types removal] Specifying types in bulk requests is deprecated.
  warnings.warn(message, category=ElasticsearchWarning)
('Unexpected error:', <class 'elasticsearch.helpers.errors.BulkIndexError'>)
5001
('Unexpected error:', <class 'elasticsearch.helpers.errors.BulkIndexError'>)
10001
('Unexpected error:', <class 'elasticsearch.helpers.errors.BulkIndexError'>)

```

The script then runs and errors with the following:

```
raise BulkIndexError("%i document(s) failed to index." % len(errors), errors)
elasticsearch.helpers.errors.BulkIndexError: ('500 document(s) failed to index.', [{u'index': {u'status': 400, u'_type': u'review', u'_index': u'anonreviews', u'error': {u'reason': u'mapper [reviewerId] cannot be changed from type [keyword] to [text]'

```

This section appears to be section in question, are you familiar with this setup or could provide any insight to get this example ported to 7.11?

```
`if len(actions) >= actionsPerBulk:
try:
    helpers.bulk(es, actions)
except:
    print ("Unexpected error:", sys.exc_info()[0])
del actions[0:len(actions)]
print (numLines)

if len(actions) > 0:
helpers.bulk(es, actions)`

```

Thanks for your help if you can provide it.

---

<div class="post-metadata">

**Author:** ![gentle\_ghost](https://avatars.discourse-cdn.com/v4/letter/g/ea5d25/32.png) [@gentle\_ghost](https://discuss.elastic.co/u/gentle_ghost)\
**Post date:** [April 16, 2021, 9:52pm UTC](https://discuss.elastic.co/t/trying-to-run-the-get-duration-by-using-bucket-script-but-not-working/269648/6 "2021-04-16T21:52:38Z")

</div>

Hello,

I appear to be making some progress but the scripts in the demo seem to have some deprecated code as the last update was 2018. I'm running into the following errors in the script and appears changes need to be made to some of the queries but not sure where. Here are the error messages:

`ElasticsearchWarning: [bool][1:94] Deprecated field [mustNot] used, expected [must_not] instead`  
`ElasticsearchWarning: [types removal] Specifying types in search requests is deprecated.`

Has anyone successfully updated ths script for 7.11?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 14, 2021, 9:53pm UTC](https://discuss.elastic.co/t/trying-to-run-the-get-duration-by-using-bucket-script-but-not-working/269648/7 "2021-05-14T21:53:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.