### Author:  [@mkorayem](https://discuss.elastic.co/u/mkorayem)
#### Post date: [March 24, 2021, 6:44pm UTC](https://discuss.elastic.co/t/google-workspace-poll-loads-of-data/268255/1 "2021-03-24T18:44:07Z")
Hi,
Using filebeat 7.12 and 7.11.2 google\_workspace does not respect the var.initial\_interval so it polls huge amount of data and also polls it multiple times confirmed from Google API dashboard too.
```auto
> |2021-03-24T20:32:54.452+0200|DEBUG|[input.httpjson-cursor]|v2/value_tpl.go:57|template execution failed: template: :1:9: executing at <.cursor.last_execution_datetime>: map has no entry for key last_execution_datetime|{input_source: https://www.googleapis.com/admin/reports/v1/activity/users/USER_AT_DOMAIN/applications/admin, input_url: https://www.googleapis.com/admin/reports/v1/activity/users/USER_AT_DOMAIN/applications/admin}|
> |---|---|---|---|---|---|
> |2021-03-24T20:32:54.452+0200|DEBUG|[input.httpjson-cursor]|v2/value_tpl.go:60|template execution: falling back to default value|{input_source: https://www.googleapis.com/admin/reports/v1/activity/users/USER_AT_DOMAIN/applications/admin, input_url: https://www.googleapis.com/admin/reports/v1/activity/users/USER_AT_DOMAIN/applications/admin}|
> |2021-03-24T20:32:54.452+0200|DEBUG|[input.httpjson-cursor]|v2/value_tpl.go:57|template execution failed: template: :1:12: executing at : wrong type for value; expected string; got time.Time|{input_source: https://www.googleapis.com/admin/reports/v1/activity/users/USER_AT_DOMAIN/applications/admin, input_url: https://www.googleapis.com/admin/reports/v1/activity/users/USER_AT_DOMAIN/applications/admin}|
> |2021-03-24T20:32:54.452+0200|DEBUG|[input.httpjson-cursor]|v2/value_tpl.go:70|template execution: evaluated template |{input_source: https://www.googleapis.com/admin/reports/v1/activity/users/USER_AT_DOMAIN/applications/admin, input_url: https://www.googleapis.com/admin/reports/v1/activity/users/USER_AT_DOMAIN/applications/admin}|
> |2021-03-24T20:32:54.452+0200|DEBUG|[input.httpjson-cursor]|v2/value_tpl.go:70|template execution: evaluated template |{input_source: https://www.googleapis.com/admin/reports/v1/activity/users/USER_AT_DOMAIN/applications/admin, input_url: https://www.googleapis.com/admin/reports/v1/activity/users/USER_AT_DOMAIN/applications/admin}|
> |2021-03-24T20:32:54.452+0200|DEBUG|[input.httpjson-cursor]|v2/value_tpl.go:57|template execution failed: template: :1:16: executing at <.last_response.body.nextPageToken>: map has no entry for key nextPageToken|{input_source: https://www.googleapis.com/admin/reports/v1/activity/users/USER_AT_DOMAIN/applications/admin, input_url: https://www.googleapis.com/admin/reports/v1/activity/users/USER_AT_DOMAIN/applications/admin}|
> |2021-03-24T20:32:54.452+0200|DEBUG|[input.httpjson-cursor]|v2/value_tpl.go:70|template execution: evaluated template |{input_source: https://www.googleapis.com/admin/reports/v1/activity/users/USER_AT_DOMAIN/applications/admin, input_url: https://www.googleapis.com/admin/reports/v1/activity/users/USER_AT_DOMAIN/applications/admin}|
```
---
### Author:  [@mkorayem](https://discuss.elastic.co/u/mkorayem)
#### Post date: [March 28, 2021, 5:23pm UTC](https://discuss.elastic.co/t/google-workspace-poll-loads-of-data/268255/2 "2021-03-28T17:23:36Z")
I tried the http-json input separately and it does not save the cursor last\_execution\_datetime
---
### Author:  [@danmcq](https://discuss.elastic.co/u/danmcq)
#### Post date: [March 31, 2021, 9:40pm UTC](https://discuss.elastic.co/t/google-workspace-poll-loads-of-data/268255/3 "2021-03-31T21:40:13Z")
I'm seeing the same behavior with a clean install of Filebeat 7.12 and the google\_workspace module.
---
### Author:  [@theherodied](https://discuss.elastic.co/u/theherodied)
#### Post date: [April 1, 2021, 1:22pm UTC](https://discuss.elastic.co/t/google-workspace-poll-loads-of-data/268255/4 "2021-04-01T13:22:17Z")
Seeing the same behaviour. Ingesting the same events thousands of times.
Had one event 16,000 times before I checked and killed the input.
---
### Author:  [@charlatan](https://discuss.elastic.co/u/charlatan)
#### Post date: [April 13, 2021, 8:19pm UTC](https://discuss.elastic.co/t/google-workspace-poll-loads-of-data/268255/5 "2021-04-13T20:19:54Z")
We're aye seeing the same thing. If you have 100,000 accounts in your domain with hundreds of millions of events over the six-month period, it can take days for a single pass to finish. We use fingerprint against the Google event ID to drop duplicates so we don't have huge indexes but the run-time is massively problematic since it doesn't respect the interval value.
---
### Author:  [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)
#### Post date: [April 13, 2021, 11:33pm UTC](https://discuss.elastic.co/t/google-workspace-poll-loads-of-data/268255/6 "2021-04-13T23:33:54Z")
Hey all! Thanks for the report on the issue, I will check this tomorrow and come back with a possible workaround while we work on a fix if we are able to reproduce the issue, seeing as many people have reported it, that should be quite likely.
The debug messages reported by @mkorayem is mostly just debug however, I do wonder if you are only getting this error once or not?:
`|2021-03-24T20:32:54.452+0200|DEBUG|[input.httpjson-cursor]|v2/value_tpl.go:57|template execution failed: template: :1:9: executing at <.cursor.last_execution_datetime>: map has no entry for key last_execution_datetime|`
The first time the beat runs, last\_execution\_datetime does not exist, because it retrieves this from the first response, and will then default back to the value you specify on `initial_interval`, however it seems that it either does not pick up the correct timestamp on the first response, or that it is stuck paginating the same page(s), which would explain the duplicate entries.
---
### Author:  [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)
#### Post date: [April 14, 2021, 11:06am UTC](https://discuss.elastic.co/t/google-workspace-poll-loads-of-data/268255/7 "2021-04-14T11:06:10Z")
Hey all, just to confirm, the fixes for this has been merged for 7.12.1 and 7.13, if anyone would want I can also provide a snapshot build of 7.12.1 if needed to test.
The relevant PRs:
>
>
> This is an automatic backport of pull request #24967 done by \[Mergify\](https://mergify.io).
>
> \---
>
>
> \
> \Mergify commands and options\
>
> \
>
> More conditions and actions can be found in the \[documentation\](https://docs.mergify.io/).
>
> You can also trigger Mergify actions by commenting on this pull request:
>
> \- \`@Mergifyio refresh\` will re-evaluate the rules
> \- \`@Mergifyio rebase\` will rebase this PR on its base branch
> \- \`@Mergifyio update\` will merge the base branch into this PR
> \- \`@Mergifyio backport \\` will backport this PR on \`\\` branch
>
> Additionally, on Mergify \[dashboard\](https://dashboard.mergify.io/) you can:
>
> \- look at your merge queues
> \- generate the Mergify configuration with the config editor.
>
> Finally, you can contact us on https://mergify.io/
> \
>
>
> This is an automatic backport of pull request #24967 done by \[Mergify\](https://mergify.io).
>
> \---
>
>
> \
> \Mergify commands and options\
>
> \
>
> More conditions and actions can be found in the \[documentation\](https://docs.mergify.io/).
>
> You can also trigger Mergify actions by commenting on this pull request:
>
> \- \`@Mergifyio refresh\` will re-evaluate the rules
> \- \`@Mergifyio rebase\` will rebase this PR on its base branch
> \- \`@Mergifyio update\` will merge the base branch into this PR
> \- \`@Mergifyio backport \\` will backport this PR on \`\\` branch
>
> Additionally, on Mergify \[dashboard\](https://dashboard.mergify.io/) you can:
>
> \- look at your merge queues
> \- generate the Mergify configuration with the config editor.
>
> Finally, you can contact us on https://mergify.io/
> \
---
### Author:  [@LiamSennitt](https://discuss.elastic.co/u/LiamSennitt)
#### Post date: [April 21, 2021, 8:07am UTC](https://discuss.elastic.co/t/google-workspace-poll-loads-of-data/268255/8 "2021-04-21T08:07:17Z")
Yes, a snapshot build of 7.12.1 would be great.
Unless there is an earlier version of filebeat which has both the `google_workspace` and `okta` modules without this issue?
---
### Author:  [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)
#### Post date: [April 24, 2021, 8:29am UTC](https://discuss.elastic.co/t/google-workspace-poll-loads-of-data/268255/9 "2021-04-24T08:29:51Z")
Hi @LiamSennitt, I think the official one shouldn't be too far away now, if that is okay 🙂
---
### Author:  [@charlatan](https://discuss.elastic.co/u/charlatan)
#### Post date: [April 30, 2021, 8:00pm UTC](https://discuss.elastic.co/t/google-workspace-poll-loads-of-data/268255/10 "2021-04-30T20:00:49Z")
Just a follow-up to this: the official build is available as of the 27th.
It's still not quite working correctly for us, I'll open a report in a new thread.
---
### Author:  [@theherodied](https://discuss.elastic.co/u/theherodied)
#### Post date: [May 1, 2021, 7:41pm UTC](https://discuss.elastic.co/t/google-workspace-poll-loads-of-data/268255/11 "2021-05-01T19:41:11Z")
Please let us know the link to the new thread so I can follow it. Haven't had a chance to try the new version yet.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 29, 2021, 9:41pm UTC](https://discuss.elastic.co/t/google-workspace-poll-loads-of-data/268255/12 "2021-05-29T21:41:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.